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Abstract 

Two problems the Aeronautical Systems Center’s Acquisition Cost Division 
(ASC/FMC) is encountering with its Life Cycle Cost/Lean Process Initiative (LCC/LPI) 
efforts are (Marshall and Seibel, 2000): (1) a high proportion of inexperienced to 
experienced cost analysts which makes access to valuable expertise limited, and (2) 
knowledge loss due to turnover of experienced cost analysts. What is needed is a 
“system that enables organizations to capture, analyze, share, apply, and reuse 
knowledge” (Cho et al, 2000:2-6). 

Using a Knowledge Management framework presented by Cho and colleagues, 
this study will demonstrate a process to generate, organize, and develop expert 
knowledge as a means to minimize knowledge loss due to turnover. The methodology 
presented in this thesis is a four-step, tailored approach to identify tasks or processes 
important to the functioning of an organization, capture knowledge from experts 
pertaining to those tasks (generate content), convert that knowledge into a flowchart 
(organize content), and have experts critique the end product to ensure accuracy and 
usefulness (develop content). 

The methodology capitalizes on proven knowledge elicitation techniques for the 
generation of knowledge and a commercial-off-the-shelf software program, Microsoft© 
Excel, for the organization and representation of knowledge in the form of a flowchart. 
The methodology is demonstrated on the process of crosschecking cost estimates and 
resulted in the creation of a procedural guide. This guide contains a flowchart 
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representing the experts’ approach to crosschecks, and hyperlinks to detailed knowledge 
sheets regarding each step. 



KNOWLEDGE MANAGEMENT IN THE COST ANALYSIS KNOWLEDGE 


DOMAIN: GENERATING, ORGANIZING, AND DEVELOPING KNOWLEGE FOR 

CROSSCHECKING COST ESTIMATES 


I. Introduction 


Background 

Today’s acquisition environment is one of shrinking and unstable budgets, rapidly 
advancing technology, and high personnel turnover in already undermanned program 
offices. Essentially, today’s acquisition workforce is forced to do more with less (Cho et 
al, 2000:1-1). An article written in 1994 by Mrs. Colleen Preston, then the Deputy Under 
Secretary of Defense (Acquisition Reform), captures many aspects of the acquisition 
environment that still apply today: 

The world in which the DoD must operate has changed beyond the limits of the 
existing acquisition system’s ability to adjust or evolve. It is not enough to 
improve the existing system.. .we must be able to procure state-of-the-art 
technology and products, rapidly, from reliable suppliers who utilize the latest 
manufacturing and management techniques; assist United States companies now 
predominantly dependent on DoD business to transition to dual-use production; 
aid in the transfer of military technology to the commercial sector; and, preserve 
defense-unique core capabilities. (Preston, 1994:8) 

Ms. Preston further states that this reality coupled with fiscal constraints makes the 

current way of conducting acquisitions unaffordable and inefficient (Preston, 1994:8). 

As a result, the acquisition community must increase its productivity while 

simultaneously reducing costs associated with procuring weapon systems. 

In order for the Acquisition Workforce to meet the needs outlined by Ms. Preston 

and succeed in an information driven business environment with scarce financial and 
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personnel resources, Cho and colleagues insist that they must “operate smarter” (Cho et 
al, 2000:1-1). To do so, however, requires a process to “take the collective ‘smarts,’ the 
knowledge and experience of the acquisition workforce, and apply it intelligently across 
the entire DoD acquisition community” (Cho et al, 2000:1-1). One of the latest 
management techniques adopted by several commercial firms to operate smarter and 
succeed in today’s rapidly changing business environment is Knowledge Management. 

Knowledge Management (KM) is “the way organizations generate, communicate, 
and leverage its intellectual assets” (President et al, 1998). According to Cho and 
colleagues, this is accomplished by “developing a framework or system that enables 
organizations to capture, analyze, share, apply, and reuse knowledge” (Cho et al, 2000:2- 
6). Cho and colleagues argue that embracing KM will allow the acquisition community 
to “see the greatest improvements in productivity” (Cho et al, 2000:1-2). 

Within the Air Force’s acquisition workforce, the cost analysis community is one 
such organization that relies heavily on knowledge to successfully accomplish its 
mission. According to Drucker, a knowledge worker is an individual who knows how to 
allocate knowledge to productive use (Drucker, 1991:71-72). Cost analysts are 
knowledge workers that use available information (cost reports, mathematical formulas, 
engineering specifications, commercial prices, etc.) to produce reliable and consistent 
cost estimates and analyses to support Air Force acquisitions. 

The Aeronautical Systems Center’s Acquisition Cost Division (ASC/FMC) is 
actively engaged in an initiative to redesign the way weapon systems’ life cycle cost 
estimates are prepared. This initiative seeks to maintain ASC’s core competency in 
preparing cost analyses and estimates given limited resources. The initiative, formally 
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called the Life Cycle Cost/Lean Process Initiative (LCC/LPI), has three goals (Marshall, 
2000): (1) reduce cycle time of the life cycle cost estimating process by 50% while 
maintaining quality; (2) revitalize the disciplined process of preparing Program Office 
Estimates; and (3) reduce process cycle time in terms of man-hours so that estimates can 
be completed within available resources. 

Problem Statement 

Two problems ASC/FMC is encountering with its LCC/LPI efforts are (Marshall 

and Seibel, 2000): (1) a high proportion of inexperienced to experienced cost analysts 

which makes access to valuable expertise limited, and (2) knowledge loss due to turnover 

of experienced cost analysts. The loss of knowledge experienced by ASC/FMC is not 

unique to Wright-Patterson AFB—it is common across the entire cost analysis career 

field. For example, the Air Force Cost Analysis Agency (AFCAA) is experiencing a 

significant amount of knowledge loss due to turnover. According to the 1998-1999 

Deputy Assistant Secretary of Cost and Economics Annual Report: 

During 1998-1999, AFCAA saw a continual decrease in personnel numbers, 
especially on the military side. The shrinking numbers are a familiar occurrence 
across the entire Air Force as the cost analysis career field has become absorbed 
into the overall financial management career field. (Deputy, 1999:53) 
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AFCAA Personnel Changes 



Figure 1. AFCAA Personnel Changes (Deputy, 1999:53) 


This turnover experienced across the cost analysis career field is a problem 
because when an individual leaves an organization, as is the case in retirement or re¬ 
assignment, the knowledge trapped within that individual departs with him or her. 
Several studies have identified the correlation between turnover and decreases in 
organizational performance (Carley, 1992; Kim, 1994; Huber, 1991). In fact, when 
studying the impact of turnover on organizational performance Carley found that 
“turnover reduces organizational performance because portions of the institution’s 
memory leave as personnel leave” (Carley, 1992:33). She also found that “organizations 
learn slower and less the higher the turnover rate” (Carley, 1992:33). 

As a result, “the knowledge trapped within the employee base must be leveraged 
to the organizational level, where it can be accessed, synthesized, augmented, and 
deployed for the benefit of all” (Marshall et. al, 1996:80) in a resource efficient manner. 
As discussed by Cho and colleagues, this requires an organization to have some type of 
KM system in place to capture as much of this knowledge as possible. Five consulting 
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firms briefed the AFCAA during the winter of 2000 on KM systems that they offer to 
clients. While these briefings were informative as to the options available to the cost 
analysis community, they provided little in terms of advancing KM efforts and providing 
an affordable solution to the knowledge problem (Jones, 2000). 

Research Objectives 

In order to counter the negative effects of turnover and achieve the goals of the 
LCC/LPI given limited financial resources, ASC/FMC must internalize KM. Cho and 
colleagues present a KM framework that consists of four main components: strategic 
plan, processes, projects, and performance measures (Cho et al, 2000:4-2). This 
framework can be operationalized by an organization to support a specific strategy, such 
as LCC/LPI, and achieve goals such as increased performance and cycle time reduction. 
This type of in-house approach provides an organization with a low-cost alternative to 
contracting out KM efforts to consulting firms. 

This study uses the LCC/LPI as a KM strategic plan and then demonstrates KM 
processes to generate, organize, and develop expert knowledge pertaining to a cost 
analysis function. This process is designed to be used by inexperienced personnel within 
a cost analysis organization to support projects such as LCC/LPI, or other strategies that 
involve knowledge. This study has two main objectives: 

1. Demonstrate a low-cost process for generating, organizing, and developing 
knowledge pertaining to a representative task in the cost analysis knowledge 
domain to support internal KM efforts. 

2. Validate the resulting knowledge representations via expert critiques. 
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Methodology 


The method chosen to satisfy the research objectives is a qualitative approach 
rooted in the concepts of KM. The methodology, which I refer to as the IC3 
methodology, is a four-step, tailored approach that allows the researcher to identify a task 
requiring expert knowledge, capture knowledge pertaining to that task, convert the 
captured knowledge into usable representations, and ensure the final product is accurate 
and useful through expert critiques. This operationalization of the KM processes 
component of the framework presented by Cho and colleagues is based on proven 
knowledge elicitation and representation techniques. 

The knowledge elicitation portion of the methodology combines the thought 
provoking ability of the think aloud method with the clarification resulting from follow¬ 
up focused discussions and semi-structured interviews. The experts for this study were 
identified by the sponsoring organization - ASC/FMC. The experts were then evaluated 
based on criteria suggested by various literature sources to determine suitability for this 
study. The IC3 method does not require the knowledge researcher to possess extensive 
knowledge of the knowledge domain nor receive special training in the utilization of the 
method. The methodology also does not require the user in the field to possess special 
software to represent the captured knowledge. 

Scope and Limitations 

The scope of this study is limited to the cost analysis knowledge domain. The 
methodology used in this study was developed specifically for this knowledge domain in 
response to the problems encountered by the sponsoring organization. It operationalizes 
the second component of the framework presented by Cho and colleagues, KM processes, 
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using the LCC/LPI as the KM strategic plan. In addition, within this domain only one 
particular task was explored to demonstrate the process. The task of crosschecking cost 
estimates was determined to be a crucial step in the cost estimating process and, 
therefore, a good representative task to use in the demonstration of the methodology 
presented in this study. As a result, the process was used to develop a procedural guide 
for crosschecking cost estimates. 

The external validity of this study is limited due to the fact that only expert cost 
analysts from Wright-Patterson AFB were interviewed. As a result, the findings of this 
study should be extrapolated to populations outside of Wright-Patterson AFB with 
caution. It is hoped, however, that the methodology presented in this study can be used 
in other locations within the Air Force cost analysis career field and produce similar 
results. 

Anticipated Contributions 

This study establishes the ability to capture and organize expert knowledge in the 
cost analysis knowledge domain. This suggests that a KM system can indeed be adopted 
in the Air Force cost analysis career field to improve productivity and minimize 
knowledge loss. This study will be useful to decision-makers considering an internal KM 
system by demonstrating a process for converting expert knowledge into usable 
knowledge representations. This will provide a benchmark to which other KM systems 
being considered can be compared. 

In addition, it is hoped that this study will begin a research stream that will further 
develop the application of KM to the Air Force financial management community. 
Finally, this study will produce a procedural guide pertaining to the performance of one 
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particular task within cost analysis. It is hoped that this guide will improve the 
productivity of non-expert cost analysts, prove effective at minimizing knowledge loss 
due to turnover, and ultimately contribute to the success of ASC/FMC’s LCC/LPI 
project. 
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II. Literature Review 


Overview 

This chapter presents information from existing literature on psychology, 
Knowledge Management, organizational learning, management, and knowledge 
acquisition and is meant to serve five main purposes. First, the terms data, information, 
and knowledge are differentiated to provide a solid understanding of knowledge and a 
working definition for this study. Second, the importance of knowledge in today’s 
business environment and Department of Defense (DoD) acquisition environment is 
discussed. Once the importance of knowledge is explored, the keepers of this valuable 
knowledge, referred to as “knowledge workers” (Drucker, 1991) , are identified and their 
importance in an organization discussed. 

Third, a Knowledge Management (KM) framework suggested by Cho and 
colleagues is presented as a way to leverage the knowledge of knowledge workers in an 
organization. This framework consists of four areas: a KM strategic plan, KM processes, 
KM projects, and KM performance measures. In this discussion, the focus of this study 
is identified as a KM process to generate, organize, and develop knowledge to support 
ASC/FMC in its LCC/LPI project. 

Fourth, some of the various techniques available for the elicitation of knowledge 
are presented. The technique(s) selected to generate knowledge for a KM system must be 
appropriate given the characteristics of the knowledge domain, limitations of the 
technique, and constraints on the researcher. As a result, a comparative analysis is 
conducted to facilitate the selection of a technique for the elicitation of knowledge. 
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Finally, knowledge representation techniques to organize knowledge generated from 
knowledge elicitation sessions are explored. A comparative analysis is then conducted to 
select a knowledge representation technique. 

Data, Information, and Knowledge 

The Merriam-Webster Dictionary defines knowledge as an “understanding gained 
by actual experience,” and “something learned and kept in the mind” (Merriam-Webster, 
1997:415). Phillipe Baumard identifies knowledge as “the object of a continuum that 
extends from interpreted information (such as a penciled diagram) to the non- 
representable (premonitions, for example)" (Baumard, 1999:19). Baumard points out that 
positivistic sciences define knowledge as “a model of reality, a sound representation of 
the world, tested and validated against the real; objective in the sense that it is 
independent of people” (Baumard, 1999:53). 

The term knowledge is often used interchangeably with the terms data and 
information, but has a very different meaning. Cho and colleagues suggest viewing the 
three terms as a continuum. 
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Figure 2 presents a visual representation of this continuum and suggests that depth 
of meaning increases as data moves toward knowledge. Cho and colleagues define data 
as “a set of discrete, objective facts commonly seen in structured records of transactions” 
(Cho et al, 2000:2-4). It is captured by either machines or people and is simply a record 
of an event, such as costs incurred in a purchase. Large collections of data are referred to 
as databases and are commonly electronic in nature. One type of data often used in DoD 
acquisitions is cost data collected from defense contractors. This data results from 
computer systems that capture raw figures of costs incurred as a contractor performs 
work on a government contract. 
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When this unstructured data is organized in some type of meaningful way, it 
becomes information (Cho et al, 2000:2-4). Drucker identifies information as “data 
endowed with relevance and purpose” (Drucker, 1998:5). Information “increases depth 
of meaning to the receiver” by revealing a pattern or trend or by providing some type of 
interpretation (Cho et al, 2000:2-4). Unlike data, which is often collected and stored in a 
database, information is mobile and found in many forms including research reports, 
textbooks, and the World Wide Web (Cho et al, 2000:2-4). The cost data mentioned 
above becomes information when a financial analyst organizes it into Earned Value (EV) 
reports. When grouped under the different EV terminology, the cost figures become 
meaningful as they provide insight into the meaning of the raw cost data. 

Finally, when individuals use information to make decisions or perform some 
type of task, knowledge is created. In the data to knowledge continuum, “converting data 
into information thus requires knowledge” (Drucker, 1998:5). Knowledge is highly 
specialized and often found in the minds of individuals (Drucker, 1998:5-6). To provide 
a working definition for purposes of this study, knowledge enables individuals to solve 
problems, perform tasks, understand related subject matter, and is of value to an 
organization because of the high level of reliable output it produces (Nonaka and 
Takeuchi, 1995; Quinn et al, 1996; Baumard, 1999; Drucker, 1992). The knowledge a 
person possesses often increases with experience and makes them an expert in their 
respective knowledge domain. Carrying the previous example forward, knowledge 
allows a decision maker to interpret the EV reports in meaningful ways. It also allows 
them to draw conclusions about the performance of the contractor based on whether 
variances are negative or positive and on other trends present in the reports. 
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Importance of Knowledge 


According to Drucker, “to remain competitive—maybe even survive—businesses 
will have to convert themselves into organizations of knowledge specialists” (Drucker, 
1998:11). He emphasizes the extreme importance of knowledge as “the primary resource 
for individuals and the economy overall” (Drucker, 1992:95). Drucker feels that if 
organizations are to become more productive, then they must work smarter (Drucker, 
1991:72). According to Drucker, “this means working more productively without 
working harder or longer” (Drucker, 1991:72). Knowledge is the tool that allows 
organizations to work smarter and hopefully more productively (Drucker, 1991:71-72). 

Quinn, Anderson, and Finkelstein also identify the source of a corporation’s 
success and competitive advantage as its knowledge base (Quinn et al, 1996:71). Quinn 
and colleagues assert that the knowledge of professionals “creates most of the value in 
the new economy” (Quinn et al, 1996:71). As a result, there is an increased emphasis for 
managers on how to properly manage this professional knowledge base to maximize the 
benefits to the organization. If managed properly, Quinn and colleagues believe that this 
professional knowledge base can produce value for an organization through activities 
such as research and development, systems management, and design (Quinn et al, 
1996:72). 

Cho and colleagues hold similar beliefs on the importance of knowledge, 
especially in DoD acquisitions. In studying private industry, they found that “the 
commercial sector already realizes that its most important competitive advantage is the 
knowledge that exists within the company” (Cho et al, 2000:1-2). In business, 
knowledge is “valued as both corporate intellectual property and a source of competitive 
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advantage” (Cho et al, 2000:2-1). Companies believe that knowledge provides them with 
a competitive advantage when the collective knowledge of the company is focused 
toward its core competency areas (Cho et al, 2000:1-2). They feel that doing so “allows 
them to bring products to market faster, at lower cost, and with greater customer 
satisfaction” (Cho et al, 2000:1-2). 

Cho and colleagues assert that in the midst of shrinking budgets and an increased 
emphasis on doing things “better, faster, and cheaper,” DoD must work “smarter” to 
successfully meet the needs imposed on it (Cho et al, 2000:1-1). Cho and colleagues 
identify the strongest assets of the Department of Defense (DoD) as “the knowledge, 
innovation, and experience of our Acquisition Workforce” (Cho et al, 2000:1-2). They 
also assert that DoD must harness its strongest assets and apply them “intelligently across 
the entire DoD acquisition community” (Cho et al, 2000:1-1). Harnessing the knowledge 
of the Acquisition Workforce in this collective manner will result in great improvements 
in productivity (Cho et al, 2000:1-2). 

The LCC/LPI efforts by ASC/FMC are an attempt to do things smarter in the cost 
analysis knowledge domain. One of the key aspects of the success of this initiative is the 
knowledge of expert cost analysts. ASC/FMC has recognized the importance of the cost 
analysis knowledge base and are seeking ways to capture and leverage this knowledge so 
that it may achieve its goals of cycle time reduction and improved quality. 

Knowledge Workers and Organizations 

Drucker states that although knowledge is a valuable resource that allows 
organizations to work smarter, it is useless unless joined with some type of task (Drucker, 
1992:95). A knowledge worker possesses the knowledge of value to an organization and 
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allocates it to productive use (Drucker, 1991:71-72). Knowledge workers own the means 
of production - knowledge - and, as a result, are valuable assets to organizations 
(Drucker, 1992:100-101). 

Although knowledge workers are able to put knowledge to productive use, the 
combination of knowledge and tasks is the responsibility of organizations. Drucker 
states: “The organization’s function is to put knowledge to work - on tools, products, and 
processes; on the design of work; on knowledge itself’ (Drucker, 1992:96). According to 
Dodgson, the firm’s strategy can directly impact the effectiveness of an organization in 
using the professional knowledge of its constituents (Dodgson, 1993:387). Dodgson 
states “firms purposefully adopt structures and strategies to encourage learning” 
(Dodgson, 1993:387). Organizations purposefully do this because they want to facilitate 
learning, which in the long run will help them achieve the goals that their strategies are 
geared towards. 

Another factor that facilitates learning, which is often a by-product of strategy, is 
organizational structure (Dodgson, 1993:388). In order for an organization to effectively 
implement its strategy, it must have the structure to support it. Dodgson points out that 
the structure of an organization defines the way individual and group-learning processes 
interact and allows the organizational learning process to stem from these interactions 
(Dodgson, 1993:388). 

The resulting relationship between knowledge workers and organizations is a 
dependent one where knowledge workers own the means of production, their professional 
knowledge, and organizations have the tools of production, tasks and problems to which 
knowledge needs to be applied (Drucker, 1992:101). As a result, the organization suffers 
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when it loses knowledge through turnover because it no longer has the means to perform 
its tasks and processes. Kathleen Carley found that when turnover occurs, “the 
organization loses the expertise and experience” of the individual(s) who are leaving 
(Carley, 1992:28). 

In order to minimize this knowledge loss, researchers stress the need for some 
type of system to capture the knowledge of individuals before they leave (Cho et al, 

2000; Carley, 1992). In order for organizations to realize the competitive advantage that 
knowledge can provide, the knowledge “must be captured, stored, and made available for 
future use” (Cho et al, 2000:2-1). Knowledge is managed by developing a framework or 
system that enables organizations to capture, analyze, share, apply, and reuse knowledge 
to improve productiveness (Cho et al, 2000:2-6). 

The Knowledge Management Framework 

While different definitions are used to define Knowledge Management (KM), the 
concept essentially involves harnessing the collective knowledge of an organization to 
increase effectiveness and efficiency. It is “the way organizations generate, 
communicate, and leverage its intellectual assets” (President et al, 1998). While 
technology has greatly enhanced our ability to manage knowledge, organizational 
members must be motivated to embrace KM in order for its benefits to be realized. This 
is especially true when the focus of the KM system is the knowledge of individuals. If 
people are not motivated to share their expertise, then the KM system will be useless. 

Cho and colleagues suggest a KM framework to use “as a starting point to 
develop a KM program that is right for your organization” (Cho et al, 2000:4-1). This 
framework consists of four main components - a strategic plan, processes for obtaining 
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and organizing knowledge, projects to further enhance the system, and performance 
measures (Cho et al, 2000:4-1). These four components are part of an iterative process 
that seeks to refine and further enhance the quality of the knowledge contained in the KM 
system. 



KM Strategic 

Plan 

* - People 
— Process 
— Technology 


( \ 

KM Performance t 

Measures 


KM Processes 

— Volume, membership, and 


— Generating Content 

usage 


~ Organizing Content 

— Organizational KM Comfort 


— Developing Content 

- Documented organizational or 
economic value 

- KM project sustainment 

KM Projects 

— Knowledge Maps 

-Virtual Collaboration -4- 

— Communities of Practice 
~ Knowledge repositories/data 
warehousing 

— Distributing Content 


Figure 3. Knowledge Management Framework (Cho et al, 2000:4-2) 


KM Strategic Plan. The strategic plan addresses areas such as people possessing 
the knowledge of interest, processes about which we are interested in capturing 
knowledge, and how technology will be used in our KM system (Cho et al, 2000:4-1). 
This step is where an organization identifies who its experts are, what tasks these experts 
are effective and efficient at performing, and how technology will help it with the overall 
development, implementation, and sharing of this knowledge. The strategic plan should 
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also address issues such as top leadership buy-in and the current culture of the 
organization, which can ultimately make or break KM efforts. 

The LCC/LPI project currently in progress at ASC/FMC can be viewed as a KM 
strategic plan. The initiative identifies the various activities and processes involved in 
compiling a life cycle cost estimate. It also recognizes the importance of having expert 
cost analysts share their knowledge with inexperienced analysts in performing many of 
these processes. Finally, it addresses the use of the Internet and ASC/FMC website as a 
means to share knowledge gathered from best practices, lessons learned, and various KM 
processes. 

KM Processes. The KM processes for obtaining and organizing knowledge are 
the tools available to obtain the necessary knowledge for the KM system. These 
processes are sequential in nature and address how content, the knowledge of experts, 
will be generated, organized, developed, and distributed (Cho et al, 2000:4-4 to 4-6). 

This study will demonstrate a KM process to generate, organize, and develop knowledge 
to support KM strategic plans such as the LCC/LPI project. 

“Generating content” involves two steps: identifying the tasks and processes that 
are vital to an organization and the experts in those vital areas, and then eliciting and 
capturing knowledge (Cho et al, 2000:4-4). In order to maximize the elicitation of 
knowledge related to the identified task, Cordingley suggests that the chosen technique 
“should suit the purpose in hand; be compatible with both the knowledge source, human 
or otherwise, and with the elicitor; and make appropriate use of the available resources” 
(Cordingley, 1989:156). 
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“Organizing content” involves representing the knowledge generated from experts 
in a way that allows it to be retrieved electronically (Cho et al, 2000:4-5). This can be 
accomplished by using a variety of commercially available knowledge-sharing tools or 
Internet software applications. Regardless of the medium chosen, “the correct 
positioning of linkages among the elements is critical to allow workers access to what 
they need when they need it” (Cho et al, 2000:4-5). This may involve conducting follow¬ 
up sessions with experts or having independent experts critique the organization of the 
content (Cho et al, 2000:4-6). 

“Developing content” is the process whereby the knowledge generated and 
organized is further refined (Cho et al, 2000:4-6). One way to do this is by having 
“recognized experts” review the content and approve it for use (Cho et al, 2000:4-5 to 4- 
6). Once this is accomplished, the content is referred to as “filtered” and is “deemed 
important, represents the best ideas of its kind, and reflects the perspective of the 
organization’s top experts” (Cho et al, 2000:4-6). Developing and organizing the content 
of a KM system often occur simultaneously because both are “collaborative functions and 
draw upon the expertise and experience of users and experts” (Cho et al, 2000:4-6). 

Finally, “distributing content” involves sharing the knowledge that was generated, 
organized, and developed in the previous steps (Cho et al, 2000:4-6). Two primary 
objectives in this step are (1) “making it easy for people to gain access to the material 
they need,” and (2) “encouraging the use and reuse of knowledge” (Cho et al, 2000:4-6). 
This step ties back in with the KM strategic plan and uses available technologies to meet 
the two objectives above. 
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KM Projects. KM projects are the various applications that will allow an 
organization to distribute and share the knowledge resulting from the KM processes. 
These projects may include knowledge maps, virtual collaboration centers, communities 
of practice, knowledge repositories, or expert systems. These projects should take 
advantage of the latest technology, such as the Internet or commercial KM software 
packages (Cho et al, 2000:4-6). Regardless of the project chosen, the goal is maximum 
access to and sharing of the knowledge contained within the system. A KM system is 
useless unless people can access, use, and benefit from the knowledge contained within 
the system. 

KM Performance Measures. KM performance measures tell an organization 
how successful the KM system is in terms of ease of use and quality of knowledge 
contained (Cho et al, 2000:4-11). These performance measures may include the volume 
of members accessing the system, the number of hits per day to a website, research 
documenting the economic value of the system, or surveys to assess the organizational 
acceptance of the KM system (Cho et al, 2000:4-11). The feedback obtained from the 
performance measures will allow you to further enhance the value and quality of the 
system, ensuring continued use and benefit from the KM system. 

Identifying Experts 

A key source of knowledge for a KM system is experienced knowledge workers, 
who are also referred to as experts. Experts must be carefully selected to ensure the 
content generated, organized, developed, and distributed is of high quality. The literature 
recommends several different guidelines and criteria for selecting experts. A good 
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starting point is to define the term expert. Hoffman and colleagues define an expert as 
follows: 

The distinguished or brilliant journeyman, highly regarded by peers, whose 
judgments are uncommonly accurate and reliable, whose performance shows 
consummate skill and economy of effort, and who can deal effectively with rare 
or “tough” cases. Also, an expert is one who has special skills or knowledge 
derived from extensive experience with subdomains (Hoffman et al, 1995:132). 

Meyer and Booker state “an expert is anyone especially knowledgeable in the 

field.. .and at the level of detail” required for the specific task of interest (Meyer and 

Booker, 1991:85). This means that just because someone is considered an expert in a 

knowledge domain, he or she is not necessarily an expert in various sub-domains of the 

overall knowledge domain. For example, just because someone is a medical doctor does 

not mean that he or she is an expert in a specialized branch of medicine, such as 

neurology. Because of this fact, care must be taken when selecting experts relevant to a 

specific task or problem (Meyer and Booker, 1991:85-86). 

Hall and colleagues recommend having a supervisor of a given office select his or 

her best workers (Hall et al, 1995:36). This provides a starting point from which experts 

can be selected. Because the supervisor interacts with these individuals on a daily basis, 

he or she has a good idea of who the experts are. However, while these individuals may 

be experts relative to that particular office, they may not necessarily be experts relative to 

the entire cost analysis career field. Hall and colleagues recognize this and point out that 

experts will indeed differ in expertise and, as a result, as many experts as possible need to 

be considered for the study (Hall et al, 1995:35-36). 

Since the key to knowledge is experience, years of experience constitute expertise 

(Nonaka and Takeuchi, 1995; Meyer and Booker, 1991; Hall et al, 1994). Hall and 
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colleagues state that experts should have anywhere from 8 to 10 years of experience in 
the knowledge domain and be “actively engaged in hands-on problem solving” (Hall et 
al, 1995:37). In addition, professional criteria such as “graduate degrees, training 
experience...memberships in professional societies, etc.” are also important 
considerations when selecting experts (Hoffman et al, 1995:131). 

Knowledge Elicitation Techniques 

In order for knowledge to be captured by a knowledge researcher, it must first be 
elicited from the experts that possess it. Knowledge elicitation “is the process by which 
facts, rules, patterns, heuristics, operations, and procedures used by human experts to 
solve problems in the particular domain” of interest are obtained (Garg-Janardan and 
Salvendy, 1990:85). It is the first, essential step in the KM processes suggested by Cho 
and colleagues. Knowledge elicitation is also important because it is elicited knowledge 
that is the basis for subsequent knowledge representations (Garg-Janardan and Salvendy, 
1990:85). In fact, knowledge elicitation is so important in this process that it is often 
referred to as the “bottleneck” in knowledge conversion processes (Diaper, 1989:24; 
Garg-Janardan and Salvendy, 1990:85). As a result, care should be taken when eliciting 
knowledge to ensure it is relevant to the specific problem of interest, useful, and complete 
(Bell and Hardiman, 1989:65). 

Researchers in various academic fields have developed advanced techniques to 
further increase our ability to elicit knowledge. A brief discussion of some of the more 
prevalent techniques found in the literature is presented here to familiarize the reader 
with the various techniques available. After the techniques are discussed, a comparative 
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analysis is conducted to determine the most suitable technique to elicit the knowledge 
pertaining to the focus of this study. 

Structured Interview. A structured interview is an interview where the 
interviewer asks the same questions in the same order in every interview conducted. This 
type of knowledge elicitation technique is used when the wording of responses is 
important and when consistency across interviewees is important. When formulating the 
questions for a structured interview, the interviewer should be careful to avoid leading 
questions or those that may limit the response of the expert (Cordingley, 1989:114). 

Semi-structured Interview. Semi-structured interviews are similar to structured 
interviews in that the researcher has a prepared list of questions to ask each expert. The 
main difference between a structured and semi-structured interview is that the order in 
which questions are asked and the exact wording of the questions varies from interview 
to interview (Cordingley, 1989:114). This gives the researcher latitude to spend more 
time on topics deemed important or that the particular expert being interviewed has 
special expertise. This flexibility can pose a challenge to the researcher because he or she 
must be careful not to repeat questions. This interview technique can allow the expert 
being interviewed to talk more freely about topics because he or she does not have to 
follow a structured format (Cordingley, 1989:114). As a result, the semi-structured 
interviews will flow more smoothly than structured interviews (Cordingley, 1989:114). 

Focused Discussion. This technique is similar to interviews in that it seeks to 
elicit knowledge in verbal form, rather than demonstration of a physical task. It is 
characterized as “introspective” due to the fact that it forces the expert to think critically 
about a subject specified by the knowledge researcher such as steps to solve a problem or 
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perform a task (Cordingley, 1989:117-118). The main difference between interviews and 
focused discussions is that the focus is on a specified task or problem within the 
knowledge domain of interest, rather than specific questions. Focusing on a certain area 
like this removes the barrier that may exist between the interviewer and the expert and 
allows the expert to focus on the task at hand, and the interviewer to focus on how the 
expert handles the specified task or problem (Cordingley, 1989:117-118). 

Teachback. This knowledge elicitation technique requires the researcher to teach 
some part of previously elicited knowledge back to the expert. The knowledge to be used 
for the teaching part of this process may come from any technique described in this 
section. The teachback session may occur at the end of the initial knowledge elicitation 
session, or in a follow-up session. The benefit of this technique is that experts have a 
chance to “review their expertise from the outside” (Cordingley, 1989:124). This allows 
them to provide further interpretation of the elicited knowledge, have buy-in in the 
knowledge elicitation process, and build rapport with the knowledge researcher 
(Cordingley, 1989:124). 

PARI (Precursor to action, Action, Result, Interpretation) Method. This 
technique, originally developed for electronic troubleshooting, is a cognitive task analysis 
method that examines “cognitive tasks that involve the interaction of a human problem 
solver with a complex system” (Hall et al, 1994:32). The method is a nine-step process, 
includes structured interviews of several experts, and does not require the knowledge 
researcher to have extensive knowledge of the domain of interest. The main goal of this 
method is to understand how experts solve complex problems and essentially what 
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knowledge they use to do so (Hall et al, 1994:3-34). This technique is very involved and 
therefore can be time consuming. 

Knowledge Acquisition Software Tools. This type of technique utilizes 
computer software to elicit knowledge from experts. These types of tools prompt experts 
to input answers to questions and then structures that input in a way that is meaningful to 
the end user. This approach to knowledge elicitation can be very expensive since you 
must either develop or purchase the knowledge acquisition software, and then purchase 
the equipment to run it on. Two examples of knowledge acquisition software are the 
ASK method and MOLE. 

The ASK method utilizes a computer software program, referred to as the 
“knowledge acquisition assistant (KAA),” to elicit knowledge from experts and consists 
of three components (Gruber, 1990:122). The first component is the “justification 
language,” which allows experts to explain why they make the decisions they do in the 
completion of a given task (Gruber, 1990:122). This component is supported in the KAA 
and allows an expert to specify the steps taken to complete a task and justification for the 
steps taken. 

The second component is the “operational representation of strategic knowledge” 
in the form of “strategy rules” (Gruber, 1990:124). These rules explicitly state which 
steps were taken, and why they were taken, to complete a given task. The rules also 
allow the KAA to make recommendations on what steps to take in various situations. 
Finally, the third component of the ASK method is the method for transforming the 
elicited knowledge into the rules used in the second component. This component is the 
process by which the KAA transforms the knowledge taken from experts (component 1) 
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into the strategy rules available for other users in different situations (component 2) 
(Gruber, 1990:127). 

The second example of Knowledge Acquisition Software, MOLE, is useful when 
the task of interest is diagnostic in nature and involves a high degree of uncertainty 
(Eshelman, 1990:204-205). With this method, experts input knowledge into MOLE 
regarding steps taken in the diagnosis of some illness or problem. The resulting expert 
system uses a problem solving method that consists of three knowledge roles. First, for 
each symptom or event identified by the user, a set of “alternatives or possible 
explanations” is generated (Eshelman, 1990:205). Second, the user is required to input 
more information that helps MOLE “differentiate among these alternatives” (Eshelman, 
1990:205). Finally, the resulting explanations are combined into a form that is 
understandable and usable to the user (Eshelman, 1990:205). 

Critiquing. This technique involves having one expert critique the way another 
expert handles a problem or task. Although an expert can be asked to critique his or her 
own process, little knowledge will be gained due to the fact that experts are rarely aware 
of their own misunderstandings of a knowledge domain. Critiquing requires the 
knowledge researcher to have a fairly thorough understanding of the knowledge domain 
of interest, which in turn allows him or her to be aware of alternatives and understand the 
approach taken by an expert. The knowledge researcher explains the steps taken by 
another expert to the current expert and seeks both positive and negative feedback. This 
additional feedback both improves the quality and increases the amount of knowledge 
elicited (Cordingley, 1989:139-140). 
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Role-Play. Role-playing requires having the expert assume a role in a situation 
frequently encountered in the knowledge domain of interest. This technique requires one 
other person to assist in performing the role-play. The situation is created by the 
knowledge researcher, who then provides details to those involved in the role-play. The 
role-play is then observed by the knowledge researcher and may be recorded on 
audiotapes or by video for later reference. Immediately following the role-play, the 
knowledge researcher should “de-brief’ the expert involved to gain additional insight into 
the knowledge used to work through the situation (Cordingley, 1989:146). 

Simulations. A simulation is similar to role-playing in that it uses an artificial 
situation to elicit knowledge from an expert. It is different than role-play because the 
situation is “treated as real as far as the performance is concerned” (Cordingley, 
1989:146). Simulations are often very expensive and time consuming to set up and run 
and, as a result, are often not practical to use for knowledge elicitation (Cordingley, 
1989:146). 

Verbal Protocol Analysis. Verbal protocol analysis is a broad term that 
describes several techniques all designed to capture and document the knowledge used by 
experts to solve a problem or complete a task (Cordingley, 1989; Hall et al, 1994). These 
techniques require experts to think or talk aloud while performing a task, which in turn 
elicits the knowledge employed to do so. Due to the large amount of documentation that 
the knowledge researcher may have to do, it is recommended that audio taping be used to 
ensure that all important elicited knowledge can be captured (Cordingley, 1989:100). 

Some of the more popular techniques include think aloud, talk aloud, user 
dialogues, behavioral description, retrospective reports, and various types of interviews 
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(Cordingley, 1989; Hall et al, 1994; Someren et al, 1994). Of these, the think aloud 
method is perhaps the most effective (Someren et al, 1994:1-2). The think aloud method 
“has its roots in psychological research” and was developed from an older method called 
introspection (Someren et al, 1994:29). The underlying premise of this technique is that 
“one can observe events that take place in consciousness” similar to the way we observe 
things in the world around us (Someren et al, 1994:29). 

The technique requires an expert to think aloud, or verbally communicate the 
thoughts in his or her mind, as they work through a task (Cordingley, 1989; Someren et 
al, 1994). Because the technique is not structured and allows the subject to convey his or 
her thought process with little or no disturbance, the technique is “easy for subjects, 
because they are allowed to use their own language” (Someren et al, 1994:26). The 
resulting verbal protocols can then be analyzed and used to construct explicit 
representations. 

Interactive Group. This knowledge elicitation technique requires assembling 
experts in a group and having the researcher serve as the group moderator (Meyer and 
Booker, 1991:102). Once assembled, the moderator provides any degree of structure 
necessary to elicit knowledge regarding a particular task or topic of interest (Meyer and 
Booker, 1991:103). In order to avoid negative effects such as groupthink, Meyer and 
Booker suggest “a highly structured group” that is “carefully choreographed” (Meyer and 
Booker, 1991:103). This technique is fairly effective for providing predictions or 
judgment regarding risk assessment due to the “synergism” that results from expert 
interaction (Meyer and Booker, 1991:103). However, this technique is often difficult to 
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use because of the coordination between several experts’ schedules that must occur and 
the “potential for group-think bias” (Meyer and Booker, 1991:103). 

Comparative Analysis of Knowledge Elicitation Techniques 

In order to maximize the elicitation of knowledge related to a task, it is essential 
that the knowledge elicitation technique employed matches the characteristics and needs 
of that task. Cordingley suggests that the chosen technique generally “should suit the 
purpose in hand; be compatible with both the knowledge source, human or otherwise, and 
with the researcher; and make appropriate use of the available resources” (Cordingley, 
1989:156). However, a review of the literature revealed that it is difficult to compare 
techniques. 

Several studies have attempted to empirically compare and classify techniques 
based on effectiveness in various knowledge domains, but have had mixed results. 
Hoffman and colleagues reviewed several such studies and found that recommendations 
for or against techniques are often contradictory and biased (Hoffman et al, 1995:140). 

As a result, “it is difficult to make comparisons” between different knowledge elicitation 
techniques based on existing literature (Hoffman et al, 1995:140). 

The differential access hypothesis reflects the belief that different techniques 
“may elicit different types of knowledge (e.g., declarative versus procedural, explicit 
versus tacit, verbal versus perceptual, etc.) and different kinds of strategies” (Hoffman et 
al, 1995:142). Several studies have attempted to empirically prove this hypothesis, but 
have not had very much success. After conducting a review of existing literature on the 
differential access hypothesis, Hoffman and colleagues concluded that “research has 
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failed to support a strong version of the differential access hypothesis” (Hoffman et al, 
1995:145). 

Despite the apparent difficulty in making comparisons, Hoffman and colleagues 
found that “methods can differ in the relative efficiency with which they yield 
knowledge” (Hoffman et al, 1995:142). Some of the areas on which Hoffman and 
colleagues contend that techniques can be compared are materials needed, simplicity of 
the task of interest, duration of the task, uniqueness of the task, and efficiency of the 
technique (time required to use the technique) (Hoffman et al, 1995:141-143). Other 
sources recommend techniques be compared on the basis of experience required of the 
researcher, problem solving method employed in the performance of the task, and the 
nature of the knowledge domain of interest (Kitto and Boose, 1990; Cordingley, 1989). 

A comparative analysis that takes into account factors such as those identified 
above, as well as unique research constraints, is needed in this study. Overall, there are 
six constraints that need to be considered in both this study (due to the researcher’s 
limitations) and in the field for an organization internally engaged in Knowledge 
Management efforts. Within each of these six constraints, the eleven knowledge 
elicitation techniques described in this chapter need to be compared and assigned a 
relative ranking in terms of how well the technique satisfies the constraint. Table 1 
contains the six constraints under which the knowledge elicitation techniques need to be 
compared for this study. 
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Table 1. Constraints under which to evaluate knowledge elicitation techniques 


1. The technique must not require special training to use 

2. The technique must not require special equipment to use 

3. The technique must be suitable for eliciting complex problem solving expertise 

4. The technique must be easy for expert to interface with 

5. Use of the technique must not require the knowledge researcher to possess 
extensive knowledge in the knowledge domain of interest 

6. The technique must be time efficient 


Tables 2 through 7 contain the criteria for assigning relative rankings to each 
knowledge elicitation technique within each constraint when compared to other 
techniques being considered. 


Table 2. Criteria for evaluating knowledge elicitation techniques under constraint 1 


Ranking 

Criteria used to assign ranking 

✓ 

A good candidate for internal knowledge elicitation efforts. The 
instructions found in existing literature sources on how to employ this 
technique are clear, straightforward, and simple relative to other 
techniques considered. As a result, when compared to the other 
techniques this technique does not require special training in order to 
properly employ it in knowledge elicitation efforts. 

X 

Not a good candidate for internal knowledge elicitation efforts. The 
instructions on how to employ this technique are either not available in 
existing literature, are too complex to grasp, or much too time consuming 
to learn relative to other techniques considered. As a result, when 
compared to other techniques this technique requires some type of formal 
training (i.e. courses, years of academic training, mastery of special 
manuals) for its use in knowledge elicitation efforts. 

9 

• 

Caution should be exercised when considering this technique for 
internal knowledge elicitation efforts. The instructions on how to 
employ this technique may be confusing or too complicated to some users 
relative to other techniques considered. As a result, this technique may 
require some type of training (formal or informal) depending on the 
background of the knowledge researcher. 
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Table 3. Criteria for evaluating knowledge elicitation techniques under constraint 2 


1 Ranking 

Criteria used to assign ranking 

s 

A good candidate for internal knowledge elicitation efforts. When 
compared to the other techniques, this technique does not require special 
equipment for proper employment, such as unique software programs or 
electronic equipment, which may not be available to the knowledge j 

researcher for use in knowledge elicitation. 

X 

Not a good candidate for internal knowledge elicitation efforts. When 
compared to the other techniques, this technique requires special 
equipment for proper employment, such as unique software programs or 
electronic equipment, which may not be available to the knowledge 
researcher for use in knowledge elicitation. 

9 

• 

Due to the dichotomous nature of this constraint, this ranking is not j 

appropriate. ] 


Table 4. Criteria for evaluating knowledge elicitation techniques under constraint 3 


Ranking 

Criteria used to assign ranking 

s 

A good candidate for internal knowledge elicitation efforts. Based on a 
review of the literature, this technique has been identified as able to elicit 
knowledge used in problem solving activities. The cost analysis 
knowledge domain is analytical in nature and often requires knowledge to 
be used for problem solving and conflict resolution (i.e. crosschecking 
estimates, reconciling independent and program office estimates). 

X 

Not a good candidate for internal knowledge elicitation efforts. Based 
on a review of the literature, this technique is not suitable for eliciting 
knowledge used in problem solving. Rather, this technique has been 
identified as suitable for eliciting other types of knowledge such as expert 
opinion or internal beliefs. 

9 

• 

Caution should be exercised when considering this technique for 
internal knowledge elicitation efforts. Based on a review of the 
literature, this technique has been successfully used to elicit problem 
solving knowledge in certain knowledge domains, but has produced 
mixed results in others. 
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Table 5. Criteria for evaluating knowledge elicitation techniques under constraint 4 


Ranking 

Criteria used to assign ranking 

■/ 

A good candidate for internal knowledge elicitation efforts. Based on a 
review of the literature, this technique has been identified as easy for an 
expert to interface with or participate in. This is an important 
consideration in any knowledge domain because since the expert 
possesses the knowledge that we are interested in capturing, any 
resistance or difficulty they encounter in sharing may stall and even 
damage knowledge elicitation efforts. 

X 

Not a good candidate for internal knowledge elicitation efforts. Based 
on a review of the literature, this technique has been identified as difficult 
for an expert to interface with due to a variety of reasons. These reasons 
include the need for the expert to go to a special location, learn how to 
use a software program, or exert a great deal of mental energy. 

• 

Caution should be exercised when considering this technique for 
internal knowledge elicitation efforts. Based on a review of the 
literature, this technique may prove cumbersome for an expert to interface 
with depending on a variety of factors. These factors include things such 1 
as the requirement to interface with other non-expert individuals, the need 
to physically engage in some activity, or schedule constraints of the 
expert. 


Table 6. Criteria for evaluating knowledge elicitation techniques under constraint 5 


Ranking 

Criteria used to assign ranking 

✓ 

A good candidate for internal knowledge elicitation efforts. Based on a 
review of the literature, this technique involves interaction on the part of 
the knowledge researcher that does not require extensive knowledge of 
the domain of interest. Rather, it has been identified as suitable for use by 
anyone engaging in knowledge elicitation efforts into a knowledge 
domain in which they are inexperienced. 

H 

Not a good candidate for internal knowledge elicitation efforts. Based 
on a review of the literature, this technique requires the knowledge 
researcher to be knowledgeable in the domain of interest in order to 
properly employ it. 

• 

Caution should be exercised when considering this technique for 
internal knowledge elicitation efforts. Based on a review of the 
literature, this technique may require some knowledge pertaining to the 
domain of interest so that terminology can be understood, questions for 
sessions can be formulated, and elicited knowledge can be understood in 
sufficient detail to formulate knowledge representations. 
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Table 7. Criteria for evaluating knowledge elicitation techniques under constraint 6 


Ranking 

Criteria used to assign ranking 

7 

A good candidate for internal knowledge elicitation efforts. When 
compared to other techniques considered, this technique is relatively time 
efficient to employ. This is important because experts often operate * 

under time constraints that make very time consuming techniques 
prohibitive. In addition, long knowledge elicitation sessions can be very 
mentally taxing for both the expert and the knowledge researcher, causing 
attention spans to decrease. 

X 

Not a good candidate for internal knowledge elicitation efforts. When 
compared to other techniques considered, this technique is very time 
consuming to employ. 

? 

• 

Caution shoidd be exercised when considering this technique for 
internal knowledge elicitation efforts. When compared to other 
techniques considered, this technique may be time consuming depending 
on several factors. These factors include detail of the knowledge elicited, 
amount of knowledge elicited, amount of documentation required, and 
length of responses necessary to adequately answer questions or explain 
concepts. 


Now that the criteria for assigning rankings within each constraint have been 
established, table 8 contains a comparative analysis of the eleven knowledge elicitation 
techniques considered based on a review of the literature and unique constraints in this 
particular study. 
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Table 8. Comparative Analysis of Knowledge Elicitation Techniques 


Technique 

Does not 
require 
special 
training to 
use 

Does not 
require 
special 
equipment 

Suitable 

for 

complex 

problem 

solving 

expertise 

Easy for 
expert to 
interface 
with 

Does not 
require 
extensive 
knowledge 
of domain 

Time 

Efficient 

Structured 

Interview 

/ 


9 

• 

9 

• 

9 

• 

/ 

Semi- 

structured 

Interview 



9 

• 


9 

• 

9 

• 

Focused 

Discussion 





9 

• 

9 

• 

Teachback 




9 

• 

X 

9 

• 

PARI 

Method 

9 

• 





9 

• 

KA 

Software 

Tools 

X 

X 


9 

• 

X 

9 

• 

Critiquing 



9 

• 

s 


9 

• 

Role-Play 


X 

9 

• 

9 

• 

9 

• 

X 

Simulation 


X 

9 

• 

9 

• 

X 

X 

Verbal 

Protocol 

Analysis 






9 

• 

Interactive 

Group 




9 

• 

9 

• 

9 

• 


Key : ^ = meets criteria; X = does not meet criteria; ? = may present problems 


35 

































Although several methods exist for knowledge elicitation, individually they fail to 
meet the needs of this study for several reasons. First, some of the methods can be quite 
demanding in regard to time required to perform them and funds necessary to obtain 
special equipment. Second, some of the methods require the knowledge researcher to 
obtain special training. This further increases the funding and time required to 
successfully utilize the method. Third, some of the methods require special equipment 
such as software programs or video recording equipment. Finally, some of the methods 
not only require extensive knowledge about the knowledge domain of interest, but also of 
domains such as computer programming and cognitive psychology. 

All of these reasons present problems for a researcher attempting to elicit 
knowledge from experts. In addition, Hoffman and colleagues advise against relying 
upon one method for knowledge elicitation (Hoffman et al, 1995:146). Doing so, they 
believe, limits the researcher and “might provide partial information” (Hoffman et al, 
1995:146). As a result, a combination of three techniques that fulfilled the research 
constraints in sufficient capacity for this study will be used for knowledge elicitation 
efforts. They are the think aloud (a verbal protocol analysis technique), focused 
discussion, and semi-structured interview techniques. The critique method will be used 
to “filter” the content of the knowledge representations and provide validation. 
Knowledge Representation Techniques 

Once the knowledge is elicited, it must be converted into knowledge 
representations that are meaningful to the end users. According to Duce and Ringland, 
knowledge representation “simply has to do with writing down, in some language or 
communications medium, descriptions or pictures that correspond in some salient way to 
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the world or a state of the world” (Ringland and Duce, 1988:3). There are many different 
techniques and models available for representing knowledge. However, selecting one 
depends on the intended use of the knowledge representation and the nature of the 
knowledge elicited from experts (Ringland and Duce, 1988:3-4). 

According to Ringland and Duce, since “the only model we have for a working 
intelligent system which uses and represents large amounts of knowledge is the human,” 
psychological means of knowledge representation are extremely useful (Ringland and 
Duce, 1988:117). This fact, coupled with the researcher’s limitations, makes 
psychological means of representing knowledge worth examining for this study. 
Regardless of the model chosen to represent the knowledge, the main concern is that the 
user, expert, and knowledge researcher are all able to understand it (Bell and Hardiman, 
1989:74). Unless the representation of the knowledge is understandable, it is useless to 
the end user. In addition, the model should be easy for the user to interface with and 
apply to other related tasks (Bell and Hardiman, 1989; Cordingley, 1989). In order to 
familiarize the reader with the various ways to represent knowledge, this section will 
briefly discuss some techniques that are appropriate for this study. 

Cognitive Models. One way to represent knowledge is through the use of 
cognitive models (Hall et al, 1995:70). Cognitive models provide visual representations 
of the thought process an expert goes through when solving a problem (Hall et al, 
1995:70). Gott points out that although complex knowledge domains usually have 
complex manuals and technical documentation associated with them, the experts who 
operate in those domains internally represent the knowledge they use to perform work in 
“streamlined mental representations, or models” (Gott, 1994:3). An example of a 
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cognitive model that was generated by Hall and colleagues for manual avionics 
troubleshooting can be seen below in figure 4. 
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Figure 4. Cognitive model for manual avionics equipment troubleshooting (Hall et al, 

1994:85) 


Concept Maps. According to Novak, “concept maps are a knowledge 
representation tool” that present concepts from “higher order - more general - concepts 
at the top to the lower order - more specific - concepts at the bottom” (Novak, 1998:3). 
The maps also have “crosslinks” that show how different concepts are related to each 
other and provide cohesion to the model (Novak, 1998:3). A pioneer in the field of 
learning, David Ausubel, saw concept maps as a way to present new information in a 
structure that “is related to an existing relevant aspect of an individual’s knowledge 
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structure” in order to facilitate “meaningful learning” (Novak, 1998:51). While concept 


maps tend to be somewhat top-level and often simple representations of knowledge, they 


are nevertheless useful in aiding a student to leam new knowledge (Novak, 1998:60-61). 


They are useful because they visually connect specific concepts and their related 


meanings and functions to more top-level concepts as can be seen in figure 5. 


( CONCEPT 1 
HAPS J 


REPHrSENT 



Figure 5. Concept map showing key ideas and principles exhibited in a good concept map 

(Novak, 1998:32) 


According to Novak, the use of concept maps as a tool to facilitate meaningful 
learning has four important advantages. First, knowledge that is acquired in a manner 
meaningful to the user “is retained longer - much longer in many instances” (Novak, 
1998:61). Second, once knowledge is learned, “subsequent learning of related materials” 
is greatly enhanced (Novak, 1998:61). Third, even if the knowledge learned through the 
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use of concept maps is not used again, the “residual effect” facilitates learning in other 
related areas (Novak, 1998:61). Finally, knowledge that is learned in a meaningful way 
can be “applied in a wide variety of new problems or contexts; the transferability of 
knowledge is high” (Novak, 1998:61). 

Rule Sets. Representing knowledge in the form of some type of rule set puts the 
knowledge in a “readily understandable form” (Ringland and Duce, 1988:103). When 
the knowledge elicited pertains to the performance of a task or procedure, it can be 
represented as a “declarative sequence of propositions” or set of rules the user can work 
through (Ringland and Duce, 1988:124). Rule sets can either be chronological, in that 
they walk the user through various steps to solve a problem or perform a task, or 
conditional, taking the form of ‘if X, then Y’ (Ringland and Duce, 1988:124-125). 
Regardless of the form the rule set takes on, the goal is to explicitly represent the 
knowledge drawn upon by experts to perform tasks in a sequence that is meaningful to 
the end user (Someren et al, 1994; Ringland and Duce, 1988). 

Flowcharts. Another tool that can be used to represent captured knowledge is a 
flowchart. A flowchart is “a graphic illustration of the steps to follow in order to arrive at 
the solution to a problem” (Computer, 2000). It is composed of “a variety of symbols 
connected by lines and arrows” that provides the user with a “clear picture of each stage 
of the process, the interrelationship between stages of the process, and the direction of 
process flow” (Chaneski, 2000:52). An example of a flowchart that represents the 
process of making a phone call can be seen in figure 6. 
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Figure 6. Example of a flowchart that represents the process of making a phone call 


Flowcharts can be easily created in commercial software programs such as 
Microsoft© Word and Excel, making them attractive as a knowledge representation tool 
since virtually all DoD employees have desktop computers that run this software. 
Creating flowcharts using these commercial-off-the-shelf (COTS) software programs 
does not require extensive training because the help function on these software programs 
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provides step-by-step instructions on how to create a flowchart and what the various 
flowchart symbols represent. This clearly provides the average cost analyst with a 
relatively simple and time efficient way to represent knowledge that has been captured 
from expert cost analysts pertaining to a certain process or problem solving technique. 
Comparative Analysis of Knowledge Representation Techniques 

In order to select an effective knowledge representation tool, a comparative 
analysis needs to be conducted. Similar to the one conducted for knowledge elicitation 
techniques, this analysis involves assigning relative rankings to the various techniques 
considered based on how well they satisfy existing research constraints. The constraints 
present in this study are also likely to be encountered in the field by an organization 
internally engaging in Knowledge Management efforts. In addition, the criterion 
suggested by Cho and colleagues under their Knowledge Management framework needs 
to be considered. Table 9 lists these constraints. 


Table 9. Constraints under which to evaluate knowledge representation techniques 


1. The technique must not require special equipment to use 

2. The technique must not require special training to use 

3. Use of the technique must not require the knowledge researcher to possess 
extensive knowledge in the knowledge domain of interest 

4. Cho et ai criteria - Technique allows users to electronically retrieve organized 
knowledge, ensuring maximum sharing of knowledge (Cho et al, 2000:4-5 to 4-6) 


Tables 10 through 13 contain the criteria for assigning relative rankings to each 
knowledge representation technique. 
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Table 10. Criteria for evaluating knowledge representation techniques under 

constraint 1 


Ranking 

Criteria used to assign ranking 

v' 

A good candidate for internal knowledge representation. When 
compared to other techniques, this technique does not require special 
equipment for proper employment, such as unique software programs or 
electronic equipment, which may not be available to the knowledge 
researcher. 

# 

Not a good candidate for internal knowledge representation. When 
compared to other techniques, this technique requires special equipment 
for proper employment, such as unique software programs or electronic 
equipment, which may not be available to the knowledge researcher. 

9 

Due to the dichotomous nature of this constraint, this ranking is not 

• 

appropriate. 


Table 11. Criteria for evaluating knowledge representation techniques under 

constraint 2 


Ranking 

Criteria used to assign ranking 

Y 

A good candidate for internal knowledge representation. The 
instructions found in existing literature sources on how to represent 
knowledge using this technique are clear, concise, and easy to follow. As 
a result, the knowledge researcher will not require special training or 
academic background to utilize this technique for knowledge 
representation efforts. 

* 

Not a good candidate for internal knowledge representation. The 
instructions found in existing literature sources on how to represent 
knowledge using this technique are not clear, concise, or easy to follow. 

As a result, the knowledge researcher will require special training or 
academic background in areas such as logic, cognitive psychology, or 
computer programming to properly utilize this technique for knowledge 
representation efforts. 

? 

Caution should be exercised when considering this technique for 
internal knowledge representation. The instructions on how to employ 
this technique may be confusing or too complicated to some users relative 
to other techniques considered. As a result, this technique may require 
some type of training (formal or informal) depending on the background 
knowledge researcher. 
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Table 12. Criteria for evaluating knowledge representation techniques under 

constraint 3 



Criteria used to assign ranking 

v' 

A good candidate for internal knowledge representation. Based on a 
review of the literature, this technique does not require the knowledge 
researcher to possess extensive knowledge in the domain of interest to 
represent captured knowledge. 

X 

Not a good candidate for internal knowledge representation. Based on a 
review of the literature, this technique will require the knowledge 
researcher to be knowledgeable in the domain of interest to meaningfully 
represent the captured knowledge. 

9 

• 

Caution should be exercised when considering this technique for 
internal knowledge representation. Based on a review of the literature, 
this technique may require some knowledge pertaining to the domain of 
interest so that terminology can be understood, concepts can be linked, 
and various other connections between the captured knowledge and its 
representation can be made. 


Table 13. Criteria for evaluating knowledge representation techniques under 

constraint 4 


Ranking 

Criteria used to assign ranking 

s 

A good candidate for internal knowledge representation. Based on a 
review of the literature, this technique represents knowledge in a manner 
that allows it to be shared electronically with minimum recoding or data 
transfer. ! 

X 

Not a good candidate for internal knowledge representation. Based on a 
review of the literature, this technique represents knowledge in a manner 
that does not allow it to be shared electronically without recoding or data 
transfer. This recoding or data transfer involves entering a manual 
knowledge representation into some type of software program or other 
electronic medium to facilitate user access. 

9 

1 • 

Caution should be exercised when considering this technique for 
internal knowledge representation. Based on a review of the literature, 
this technique may be created electronically but if it is not, effort will be 
spent transferring it to a medium that allows it to be accessed 
electronically. 
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Now that the reader is familiar with the criteria used to evaluate the various 


knowledge representation techniques discussed, a comparative analysis is presented to 
facilitate the selection of a knowledge representation technique for use in this study. 
Table 14 contains the comparative analysis of knowledge representation techniques. 


Table 14. Comparative Analysis of Knowledge Representation Techniques 


Technique 

Does not 
require 
special 
equipment 

Does not 
require 
special 
training to 
use 

Does not 
require 
extensive 
knowledge 
of domain 

Allows for 
Electronic 
Sharing 

Cognitive 

Models 

✓ 

X 

9 

• 

X 

Concept 

Maps 

S 

X 

9 

• 

X 

Rule Sets 

S 

9 

• 

✓ 

9 

• 

Flowcharts 

/ 


✓ 

✓ 


Based on the preceding comparative analysis, the use of flowcharts to represent 
captured knowledge satisfies every constraint. Therefore, the knowledge captured during 
knowledge elicitation sessions will be represented using flowcharts. This type of tool can 
be created using commercial-off-the-shelf software that is available to all DoD 
employees, making it an even better choice for use in internal Knowledge Management 
efforts. 
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III. Methodology 


Overview 

This chapter presents a methodology that draws upon aspects of the proven 
techniques discussed in chapter 2, is relatively simple, and resource efficient to use. The 
methodology is a tailored approach based on proven knowledge elicitation and 
representation techniques. This method, which I refer to as the IC3 methodology, draws 
upon the strengths of the three proven knowledge elicitation techniques identified in 
chapter 2 to elicit and capture expert knowledge. 

Once the researcher captures the expert knowledge, he or she then converts it into 
explicit representations that can be used by non-expert cost analysts. These knowledge 
representations are then critiqued by the original set and an independent set of experts to 
ensure accuracy and usefulness. This sequence is a KM process that can be used to 
generate, organize, and develop content in support of a KM system. The IC3 method 
does not require the knowledge researcher to possess extensive knowledge of the 
knowledge domain nor receive special training in the utilization of the method. The 
methodology also does not require the user in the field to possess special software to 
represent the captured knowledge. It is hoped that the knowledge representations 
resulting from this methodology will improve the productivity of non-expert cost analysts 
and prove effective at minimizing knowledge loss due to turnover. 

The IC3 Method 

The IC3 (Identify, Capture, Convert, Critique) method is a four-step, tailored 
approach that identifies tasks requiring expert knowledge, captures knowledge from 
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experts pertaining to those tasks, converts the knowledge into explicit representations, 
and requires experts to critique the explicit representations to ensure accuracy and 
usefulness. The method is designed specifically for obtaining knowledge from experts 
pertaining to tasks that require such knowledge. Essentially, the IC3 method is an 
operationalization of the KM processes component of the KM framework presented by 
Cho and colleagues. It provides an organization with a process that can be employed 
internally to generate, organize, and develop knowledge in support of a KM system. 
Figure 7 illustrates how the IC3 methodology fits into the KM framework. As indicated 
in the illustration, the distribution process is handled by the organization implementing 
the KM system. 



Figure 7. Illustration showing how the IC3 methodology fits into the Knowledge 

Management framework 
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(1) Identify Tasks Requiring Expert Knowledge 


Before knowledge can be captured, an organization needs to identify the tasks and 
processes it performs that require the use of expert knowledge (Scott et al, 1991:15). 
Although the Knowledge Management strategic plan should identify these processes, this 
may not always be the case. As a result, other means of identifying tasks may need to be 
used. Scott and colleagues suggest “any task for which performance improves 
significantly with experience is a good candidate” (Scott et al, 1991:15). 

In order to be able to identify candidate tasks, the researcher needs to become 
“sufficiently familiar” with the functioning of an organization and the tasks it performs 
(Scott et al, 1991:15). If the researcher is unfamiliar with the knowledge domain of 
interest, he or she can talk to experts or review some literature pertaining to the 
knowledge domain (Scott et al, 1991:15-16). The goal with all of these approaches is to 
identify the tasks or problems in the knowledge domain that are essential to the 
organization. Since this step will drive subsequent efforts to capture and convert 
knowledge, it is crucial that sufficient effort be spent on this portion of the process 
(Wilson, 1989:197-198). The more rigorous the efforts to identify tasks, the more 
effective the explicit representations will be in meeting the user requirements 
(Cordingley, 1989:98-99). 

(2) Capture Knowledge Pertaining to the Identified Task (Generate Content) 

The second step of the IC3 method captures the knowledge to be used for explicit 
representations from experts in the area of interest. This is achieved via a two-phase 
knowledge elicitation process that combines the think aloud, focused discussion, and 
semi-structured interview techniques. The first technique is initially employed to elicit 
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the first “cut” of knowledge, whereas the second two are employed in follow-up sessions 
to refine the knowledge elicited in the first session and to elicit more knowledge. 

The think aloud technique is a verbal protocol analysis technique that requires an 
expert to “relate internal thoughts” by verbally communicating them as he or she works 
through a problem (Cordingley, 1989:141). The technique is intended to “give the 
elicitor access to the thought process” (Cordingley, 1989:141) of the expert and elicit the 
knowledge that he or she uses along the way (Garg-Janardan and Salvendy, 1990:86). 
This technique is extremely useful because it allows experts to freely verbalize their 
thoughts with few constraints from the researcher, providing insight into their cognitive 
processes (Hall et al, 1994:29; Someren et al, 1994:29-31). 

The focused discussion technique requires the knowledge researcher and expert to 
engage in a follow-up conversation to “reconcile the elicitor’s understanding” about the 
knowledge elicited (Cordingley, 1989:118). It also serves to identify gaps in the 
knowledge researcher’s documentation, provide further insight into the knowledge 
elicited, and assist the knowledge researcher in structuring the verbal data resulting from 
the original knowledge elicitation session (Cordingley, 1989:118). The semi-structured 
interview technique gives the researcher latitude to spend more time on topics deemed 
important or in which the particular expert being interviewed has special expertise. This 
interview technique can allow the expert being interviewed to talk more freely about 
topics because he or she does not have to follow a structured format (Cordingley, 
1989:114). 

Overall, each phase of this step builds upon the previous phase and is aimed at 
increasing the insight into the knowledge employed by the expert in task performance, 
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clarifying the researcher’s documentation of the event, and increasing the researcher’s 
understanding of the expert’s process. Before the techniques are discussed in detail, 
criteria for selecting a sample of experts to elicit knowledge from must be established. 

Selecting Experts. In order to maximize the amount and quality of knowledge 
captured, researchers suggest a minimum of six to eight experts be interviewed (Hall et 
al, 1995:37). Meyer and Booker recommend five to nine experts, and urge a minimum of 
five experts (Meyer and Booker, 1991:87). However, due to manning shortfalls and 
experience gaps, access to this many experts may not be possible. Therefore, knowledge 
should be captured from as many experts as possible given existing constraints. 

If the researcher is not familiar with the location of experts for the identified task, 
he or she should use the sponsoring organization or headquarters office to identify 
experts. Using this method of identifying experts is consistent with the recommendation 
made by Hall and colleagues regarding the use of a supervisor to identify experts (Hall et 
al, 1994:36). Meyer and Booker recommend a similar approach and state, “selecting 
experts who are well known and respected among peers...can lend the project greater 
credibility” (Meyer and Booker, 1991:86). 

A quick checklist that incorporates the essential criteria of an expert as discussed 
in chapter 2 is found in table 15. This checklist should be used to evaluate potential 
experts identified by a supervisor or other third party source. An individual is deemed 
suitable for knowledge elicitation efforts if he or she scores a ‘yes’ for all five criteria. 
Once experts that score a yes in all categories have agreed to participate your research 
efforts, you are ready to begin eliciting and capturing their knowledge. 
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Table 15. Checklist for selecting experts 


Yes 

No 

Criteria 



Identified by supervisor or supervisory organization as an expert in the 
desired area? 



Still actively engaged in hands-on problem solving? 



Possess at least 8 to 10 years of experience in the desired knowledge 
domain? If less than 8 years experience, still regarded by peers and/or 
supervisor or supervisory organization as an expert? 



Possess at least a bachelor’s degree in an academic discipline related to 
the desired knowledge domain? 



Completed training courses in subject matter related to the knowledge 
domain? 


Phase 1 - Think Aloud Technique (Initial Session). Before the first knowledge 
elicitation session begins, the researcher should provide instructions to the expert on how 
to think aloud as he or she works through a task (Someren, 1994:42). Cordingley 
recommends four basic instructions to ensure the experts understand what they are 
supposed to do. They are: 

Say out loud everything you are thinking from the first time you see the 
question until you give an answer; 

- talk aloud constantly; 

- don’t think about what you are going to say; 

don’t explain what you are saying (Cordingley, 1989:143). 

These suggested instructions should be incorporated into some type of instruction set to 
be read to the expert before the initial knowledge elicitation session begins. These 
instructions not only provide guidance on how to participate in the think aloud method, 
they also set the ground rules for the session and cover topics such as expected duration, 
restroom breaks, phone calls, etc. The specific instructions used in this study can be 
found in Appendix A. 
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After these instructions have been explained, the expert should work through a 
practice problem related to the knowledge domain of interest (Someren et al, 1994:42- 
43). The practice problem serves two purposes. First, it allows the expert to get 
comfortable with thinking aloud while working through a task. Second, it gets the expert 
thinking about topics within the knowledge domain of interest (Someren, 1994:42-43). 
The practice problem may come from any source pertaining to the knowledge domain 
such as journals, instructional materials, or other experts. 

Once the expert has practiced thinking aloud, he or she is given a representative 
cost estimate to work through. Someren, Barnard, and Sandberg point out three 
important considerations when selecting tasks to use with the think aloud method. First, 
the task should be “at a level of difficulty that is appropriate for the subject” (Someren et 
al, 1994:36). The task should be difficult enough to stimulate thought and not one that 
can be performed in an “automated” manner (Someren et al, 1994:36). 

Second, the task should be representative of one that may be encountered in 
practice and not “unusual” just to make it difficult (Someren et al, 1994:36). In order to 
ensure the task is representative of a real-world situation, the researcher may want to 
consult other experts within the knowledge domain (Someren et al, 1994:36). Finally, 
due to time considerations, the scope of the task should be limited to a “rather small set of 
problems” (Someren et al, 1994:36). Cordingley also cites brevity of sessions as an 
important consideration when setting up a knowledge elicitation session (Cordingley, 
1989:96-97). 

While the expert thinks aloud as he or she performs the desired action on the 
representative cost estimate, the researcher takes notes on what the expert is saying. The 
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researcher should also have a copy of the cost estimate being used by the expert to 
facilitate the documentation of knowledge. This documentation can be organized using 
the structure of the cost estimate used by the expert in the think aloud session. As an 
expert thinks aloud while examining a particular part of the cost estimate, the knowledge 
researcher documents what the expert is saying next to the part of the estimate at which 
the expert is looking. For example, if the expert is thinking aloud while examining the 
hardware portion of a cost estimate, the knowledge researcher would document what the 
expert is saying next to the hardware portion of the cost estimate as shown in appendix B. 
This method of documenting and arranging elicited knowledge will facilitate knowledge 
representation efforts in the next step of the IC3 method. 

Sessions with different experts should be documented on the same note pages as 
previous sessions using color-coding as a way to distinguish between experts. The rule to 
follow when determining what knowledge to document is repetition. If a previous 
session elicited knowledge that is elicited in the current session, do not document it again. 
Only document new concepts or knowledge that is unfamiliar to the knowledge 
researcher and deemed potentially important. This will require a subjective judgment on 
the part of the researcher to determine what knowledge to document. 

When a determination is difficult to make, always err on the side of too much 
documentation rather than not enough. This will allow for a good base of knowledge to 
draw upon when organizing it into knowledge representations in the next step of the IC3 
method. The researcher should also tape record the entire knowledge elicitation session 
for future review. This ensures that all elicited knowledge is documented (Cordingley, 
1989:100-101; Scott et al, 1991:381-382). 
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Phase 2 - Focused Discussion and Semi-structured Interview (Follow-up 
Sessions). After the initial knowledge elicitation session is complete, and the researcher 
has had a chance to review his or her notes and audio recordings, follow-up sessions may 
be necessary to fill in gaps and provide further clarification (Cordingley, 1989:116-117). 
These follow-up sessions may be either focused discussions or semi-structured 
interviews. The purpose of these follow-up sessions is to remove any confusion or 
discrepancy in the researcher’s documentation, to provide the researcher further insight 
into the knowledge elicited, or simply to request assistance in organizing the written 
notes (Cordingley, 1989:116-117). Because the degree of clarification and further 
explanation depends on the background of the researcher, follow-ups should be repeated 
until the researcher is comfortable with the quantity and quality of elicited knowledge. 

As a result, the number and length of follow-up sessions may vary across researchers. 

The sessions also allow the researcher to gain further insight into the knowledge 
employed by the expert in the performance of the task and why the expert took the steps 
they did. It is very important that any questions or uncertainty be addressed in the 
follow-up sessions to ensure understanding on the part of the researcher. Knowledge 
elicited in follow-up sessions should be coded using the same process described for initial 
sessions. Once this step is complete, the knowledge of the expert is explicit to the 
knowledge researcher and is ready to be converted into knowledge representations. 

(3) Convert Knowledge Into Meaningful Representations (Organize Content) 

At this point, knowledge pertaining to the identified task has been captured and is 
now explicit to the researcher. However, it still must be made explicit to the end user. 
The third step of the IC3 method, therefore, is the conversion of the captured knowledge 


54 



into some type of knowledge representation. As indicated in chapter 2, flowcharts will be 
used as the knowledge representation technique for this study. 

Flowcharts provide a clear illustration of the flow of a process and can be easily 
created using commercial software programs such as Microsoft© Excel. This provides 
the user with a readily available tool to represent captured knowledge that allows 
maximum sharing of the knowledge, especially when the flowchart is located where it 
can be accessed those who need it. In addition, knowledge can be linked to flowchart 
symbols through the use of hyperlinks. Instructions on how to create a flowchart can be 
found in the help menu of Microsoft© Excel or by referencing existing literature on the 
topic (Computer, 2000; Lehman, 2000; Chaneski, 2000). 

To organize the knowledge elicited and captured in the previous step of the IC3 
method, both research notes and the audio recordings of the knowledge elicitation 
sessions should be used. The first step is to create a rough draft of the process. This is 
accomplished by reviewing the audio tapes and determining the general steps taken by 
each expert as he or she performed the task identified in step 1. Please note that this is a 
subjective part of the knowledge conversion process and can therefore lead to somewhat 
different representations of the process. Once the knowledge researcher determines these 
steps, he or she should represent them in a flowchart. 

Then, the elicited knowledge contained in the organized, color-coded 
documentation described in the previous step is matched with the steps of the flowchart. 
This matching of knowledge to steps in the process is accomplished using the audiotapes 
to determine where in the process the knowledge applies. Because the knowledge is 
arranged according to what point in the cost estimate it was elicited (as described in step 


55 



2), this matching process should be straightforward. For example, if one of the steps in 
the process pertains to checking calculations, using the audio tapes as a guide the 
knowledge researcher would refer to the section of the cost estimate where this step was 
accomplished. Then, the researcher would go to that point in the documentation of 
elicited knowledge and obtain the knowledge pertaining to that step. 

Once a determination has been made as to what step in the process a piece of 
knowledge applies, a hyperlink between the flowchart symbol of that step and a sheet 
containing the detailed knowledge should be created. Instructions on creating hyperlinks 
can be found in the help menu of Microsoft© Excel. Once a working rough draft of the 
flowchart with all hyperlinks has been created, the researcher should again review his or 
her notes and audio tapes to ensure all elicited knowledge is contained in the flowchart 
and that the process is consistent with that followed by the experts in the performance of 
the process. After this is accomplished, the flowchart is ready to be critiqued by experts. 
(4) Critique of Knowledge Representations by Experts (Develop Content) 

The final step of the IC3 method is a two-phase critique of the resulting flowchart 
by two sets of experts. The first group of experts to critique the flowchart should be the 
experts that were interviewed for step two of the IC3 method. The second group of 
experts to critique the flowchart should be an independent set of recognized experts in the 
knowledge domain. This process allows the original experts a chance to clarify concepts 
or correct errors in the representation before it is presented to independent experts. It also 
maintains to the relationship established between the knowledge researcher and the 
original experts by creating “buy-in” in the knowledge representation and the overall 
Knowledge Management process. 
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Not only does the critiquing step assist in the refinement and improvement of the 
flowchart, but it also contributes to the researcher’s understanding of the task and insight 
into the knowledge employed by experts in performance of the task (Cordingley, 
1989:139-140). In addition, this step serves to “filter” the knowledge content contained 
in the knowledge representation. According to Cho and colleagues, this means that 
“content has been reviewed, distilled, and approved for use by recognized experts” (Cho 
et al, 2000:4-6). 

Phase 1 - Critique by the Original Set of Experts. The first part of this step, a 
critique by the original set of experts, gives the individuals who contributed their 
knowledge to the research a chance to review how the researcher organized the 
knowledge and represented it in the flowchart. Any problems that arise or corrections 
that need to be made can be accomplished as needed. This iterative process of 
identifying and correcting discrepancies reflects the assertion by Cho and colleagues that 
organizing and developing content often “occur simultaneously” (Cho et al, 2000:4-6). 

Phase 2 - Critique by an Independent Set of Experts. Once the original set of 
experts has had a chance to review the researcher’s work and necessary corrections have 
been made, an independent set of experts critique the flowchart. The benefit of having an 
independent set of experts critique the flowchart lies in the observation by Cordingley 
that “knowledge providers will be unaware of their own domain misconceptions and 
omissions in their domain knowledge” (Cordingley, 1989:140). Independent experts, on 
the other hand, should pick up on these types of errors, if they exist, and notify the 
researcher so that the necessary corrections can be made. If problems do arise, the 
knowledge researcher takes the feedback from the independent experts and returns to the 
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previous step. Once the independent experts approve the final product, the IC3 method is 
complete. 

Once approved, the final product is ready for field-testing and initial introduction 
into the work environment. According to Harrison, “the reliability and validity” of a 
knowledge based system “must continue throughout the lifetime of the system” 

(Harrison, 1989:326). This requires a point of contact in the organization creating a KM 
system to maintain the knowledge, or content, to ensure that it is accurate and current. 
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IV. Research Findings and Analysis 


Identification of a Task Requiring Expert Knowledge 

The Knowledge Management strategic plan used for this study, the LCC/LPI, 
identified several functions and processes vital to the cost analysis and estimating 
process. One of these processes needed to be selected for the purposes of this study. The 
process of identifying potential tasks to be used for this study was a team effort between 
the researcher and the sponsoring organization. The identification step was accomplished 
over the course of two meetings with the sponsoring organization. 

During the first meeting, the researcher provided an overview of the research and 
an explanation of the criteria used to identify tasks to the sponsoring organization. This 
explanation conveyed the concept that the task should be one such that the performance 
of it “improves significantly with experience” (Scott et al, 1991:15). This allowed the 
efforts of both parties to be coordinated and centered on a common objective. In 
addition, because this step drives the efforts of the remaining steps of the methodology, 
the importance of selecting a task that meets the criteria set forth in the methodology was 
emphasized. After several possible tasks were discussed, the meeting concluded with 
both parties in agreement that more time needed to be spent on the consideration of what 
task will be used in this study. 

After both parties had a chance to evaluate the potential tasks, a second meeting 
was called to finalize the selection of a task. During this meeting, both parties agreed that 
the task of crosschecking cost estimates should be used to demonstrate the IC3 method. 
According to Mike Seibel, Chief of Cost Research for Aeronautical Systems Center, a 
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crosscheck or test of reasonableness is the process where a cost analyst “employs an 
alternative, usually simpler, methodology” to check the figures in a cost estimate (Seibel, 
2000 ). 

Essentially, the cost estimator that prepared a cost estimate ensures the cost 
figures can be backed up by alternate methodologies and that the estimate is free from 
major errors, discrepancies, or inconsistencies. For example, a cost estimator may create 
a Systems Engineering/Program Management (SE/PM) estimate by using the head count 
from a closely analogous program and multiplying it by a labor wrap rate. The 
crosscheck for the resulting SE/PM estimate could involve taking the ratio of the SE/PM 
cost estimate against the system's hardware cost and comparing the resulting percentage 
with those of other analogous programs (Seibel, 2000). According to Mr. Seibel and 
other members of the sponsoring organization, the performance of a crosscheck improves 
with experience and is an important task in cost estimating as it adds credibility to cost 
estimates. 

Capturing Knowledge From Experts Pertaining to the Selected Task 

In order to capture knowledge pertaining to the task identified in the preceding 
step, experts in the performance of that task needed to be identified and interviewed. 
Since the sponsoring organization is familiar with where the cost estimating expertise lies 
at Wright-Patterson AFB, they were instrumental in identifying experts to the researcher 
- specifically experts in crosschecking estimates. Using this method of identifying 
experts is consistent with existing literature on the subject (Hall et al, 1994:36). The 
sponsor compiled a list of experts from Wright-Patterson AFB for use in this study, and 
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seven of those experts were contacted and evaluated using the checklist provided in 
chapter 3 for participation in this study. 

These experts were all civilian employees of the Air Force and assigned to both 
program offices and cost staff positions in a variety of finance related positions. All 
experts had a minimum of 5 years experience in cost analysis positions. Since the 
majority of cost analysts at Wright-Patterson AFB are civilian employees, this sample is 
reasonably representative of the Wright-Patterson AFB cost analysis population. The 
average number of years of experience for the sample is 12.619 years. All experts 
possess at least a bachelor’s degree in a business related discipline and have completed 
Air Force, Defense Acquisition University, and various other DoD training programs in 
cost analysis and financial management. Table 16 contains the profiles of the seven 
experts interviewed. 


Table 16. Profiles of experts interviewed 


■ 

Rank 

Career 

Field 

Education 

Training 

Programs 

Years/Months 
Experience in 
Cost Analysis 

1 

GS-13 

Finance 

BA, MA 

Various AFIT, 
DAU 

5 yrs. 0 mos. 

2 

GS-14 

Finance 

BS, MBA 

Various AFIT, 
DAU 

7 yrs. 2 mos. 

3 

GS-12 

Finance 

BBA, MSA 

Various AFIT, 
DAU 

6 yrs. 0 mos. 

! 4 

GS-13 

Finance 

BBA, MBA 

Various AFIT 

13 yrs. 1 mos. 

5 

GS-14 

Finance 

BS, MBA 

Various DAU 

19 yrs. 9 mos. 

6 

GS-13 

Finance 

BA, MBA, 
MS 

Various AFIT, 
DAU 

9 yrs. 0 mos. 

7 

GM-14 

Finance 

BA 

Various AFIT, 
DAU, and DSMC 

28 yrs. 4 mos. 

| Mean years and months of experience in cost analysis 
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Phase 1 - Think Aloud (Initial Session). As discussed in chapter 3, the 
knowledge capture step is accomplished using a two-phase approach. The first phase of 
this step involves using the think aloud method to elicit and capture knowledge from the 
experts pertaining to the task of interest, which in this study is crosschecking cost 
estimates. In order to properly employ the think aloud method, two items are required. 

First, the technique requires a practice problem related to the knowledge domain 
to allow the expert to practice thinking aloud while working through a task. The practice 
problem for this study was obtained from the midterm examination for a risk 
management class, class identifier QMGT 680, taught at AFIT. The practice problem can 
be found in appendix C. This practice problem was administered at the beginning of each 
initial knowledge elicitation session. 

The second item needed for the think aloud technique is a representative task that 
the expert can think aloud while he or she works through. This need was conveyed to the 
sponsoring organization along with the criteria discussed in chapter 3 of this study. In 
response to this requirement, they provided an ACAT III program cost estimate. This 
particular estimate met the three main criteria outlined by Someren and colleagues. 

First, the scope of the estimate was such that the sponsoring organization believed 
an expert could reasonably crosscheck it within two hours. Other more detailed and 
complex estimates were considered, but were not selected due to the inordinate amount of 
time required to crosscheck them. Second, the classification level of the estimate allowed 
it to be accessed by the researcher and shared with experts. However, due to the 
proprietary information contained in the cost estimate, the name of the program to which 
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it applies and the estimate itself have been withheld in this study. Finally, the sponsoring 
organization had the cost estimate readily available for use in this study. 

Once the necessary tools had been acquired, interview sessions were scheduled 
with experts from the list provided by the sponsoring organization. The sessions were 
conducted at the experts’ offices, with the exception of one that was conducted in an 
AFIT library study room due to security restrictions at the expert’s office, to ensure 
realistic conditions and to prevent any inconvenience to the experts. Prior to beginning 
an initial session, the researcher read the instructions contained in appendix A to the 
expert. Then, the expert worked through the practice problem in appendix C until he or 
she was comfortable with the think aloud method. 

Once an expert expressed an understanding of what he or she was supposed to do, 
the session progressed to the actual performance of a crosscheck on the ACAT III cost 
estimate obtained from the sponsoring organization. A transcript of one of these think 
aloud sessions can be found in appendix D to show the reader the free-flowing nature of a 
think aloud session and the type of knowledge that might be elicited with this technique. 
Although this type of transcript does not need to be created for each interview, one was 
created to show the reader what happens in a think aloud session. Due to time 
constraints, the experts were instructed to simply indicate when they would take a time 
consuming action such as making a phone call, going to the cost library to look 
something up, gathering information from a technical expert, or gathering data to run a 
parametric model on a software program. These activities were then noted and discussed 
in follow-up sessions. 
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During the entire session, the researcher tape-recorded and took notes on the 
knowledge elicited during the session. At the conclusion of the session, the expert was 
notified that he or she might be contacted via telephone or email regarding a follow-up 
session. After all initial knowledge capture sessions with experts were complete, the 
captured knowledge contained on both the audio tapes and research notes was reviewed 
and coded using the structure of the ACAT III cost estimate briefing slides. The 
knowledge was recorded in Microsoft© Powerpoint in the note page view and color- 
coded according to what expert it came from. 

The coding of knowledge in this manner was time efficient and took advantage of 
the structure of the existing cost estimate briefing. An example of the documentation of 
elicited knowledge pertaining to crosschecking the hardware portion of the estimate can 
be seen in appendix E. As you can see in appendix E, the knowledge pertaining to a 
certain section of the cost estimate (in this example the hardware section) was 
documented next to that section. Once all initial sessions were documented using this 
color-coding method, the researcher reviewed the knowledge. Any questions or 
discrepancies that were experienced during this coding process were documented for 
follow-up sessions. 

Phase 2 - Focused Discussion (Follow-up Sessions). After the initial 
knowledge was coded, several questions remained. Some of these questions are 
contained in Appendix F, however this is not a complete list. Because of the free-flowing 
nature of the techniques utilized for the follow-up sessions, several other questions and 
topics came up during discussions with the experts that further contributed to the 
researcher’s understanding of crosschecking cost estimates. All of these questions did, 
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however, address the time-consuming steps identified in the initial session (which the 
experts were instructed to indicate but not perform due to time constraints), differences 
between expert’s approaches, and clarification of elicited knowledge. 

The experts were given three options for follow-up sessions in order to ensure that 
these sessions were conducted in a convenient manner. An email message was sent to the 
experts asking them if they preferred a follow-up session in person at their office, by 
telephone, or by email. The last option was more of a questionnaire that contained the 
questions contained in Appendix F. This option was only offered as a last resort when 
time constraints on the expert prevented them from engaging in telephone or face-to-face 
follow-up sessions. 

After the first follow-up sessions were complete, the knowledge was used to add 
to or clarify the documented knowledge from the initial sessions. This process of 
conducting a follow-up session, adding that knowledge to the rough draft, generating 
more questions, and conducting subsequent follow-up sessions was repeated twice with 
most experts. One of the experts was unavailable due to her involvement in a source 
selection, and another expert was contacted three times due to his knowledge in a 
particular area and questions surrounding knowledge provided in a previous follow-up 
session. 

Convert Captured Knowledge Into a Flowchart 

The knowledge resulting from the knowledge elicitation step of the IC3 method 
was then used to create a flowchart in Microsoft© Excel. Once the initial coding was 
complete and all captured knowledge was documented, the audiotapes were again 
reviewed to ascertain the order of the steps taken by experts in the performance of a 
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crosscheck. A rough draft of the step-by-step process was created and refined upon 
further review of the tapes. Overall, the experts did follow a similar process as they 
crosschecked the estimate. They also used similar rules of thumb to crosscheck certain 
areas of the cost estimate. 

After a rough draft representing this general process was created, the audiotapes 
and research notes were used to match the coded knowledge to steps in the crosscheck 
process. Microsoft© Excel was used to create a flowchart of the crosscheck process. 

The hyperlink function, which allows the user to click on blocks or words pertaining to a 
step in the process and access detailed information about that step, was used to organize 
the coded knowledge in multiple worksheets and provide relevant hyperlinks. These 
hyperlinks were very useful in organizing the captured knowledge in a way that allowed 
users to quickly access needed knowledge. The complete crosscheck guide developed for 
this thesis is found in Appendix G. 

Expert Critique of Crosscheck Flowchart 

The final, and perhaps most important, step of the IC3 methodology is the critique 
step. This step strengthens the internal validity for this thesis and its resulting product, 
and “filters” (Cho et al, 2000:4-6) the knowledge contained within the flowchart. Once 
the flowchart contained all the knowledge captured from step 2 and the researcher was 
comfortable with the format and content, it was ready to be critiqued by the original and 
independent sets of experts. 

The critique sessions were conducted in the experts’ offices and lasted anywhere 
from 30 to 60 minutes. In order to ensure the experts had ample time to review the 
flowchart, the researcher emailed the Excel file to them at least one day in advance of the 
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scheduled meeting. This allowed the experts to review the flowchart at their convenience 
and write down any questions or feedback they might have. All experts were instructed 
to review the flowchart to ensure the flow of the process was correct, ensure the content 
of the knowledge was accurate and complete, and assess the usefulness of the product. 

Phase 1 - Critique by the Original Set of Experts. For the first phase of the 
critique sessions, six of the original seven experts were contacted. The one expert that 
did not participate was engaged in a source selection and, as a result, was unavailable for 
this step. The critique sessions with the original experts served to identify discrepancies 
and elicit more knowledge. For example, in one of the critique sessions an expert offered 
an additional rule of thumb for crosschecking hardware elements of cost estimates. This 
rule of thumb had not been previously suggested by any of the other six experts. 

Another expert suggested ways to make the flowchart more user friendly and, 
hopefully, a better tool for inexperienced cost analysts. This included spelling out all 
acronyms, providing more examples, and more links to existing knowledge sources on 
the Internet. All changes and improvements that needed to be made as a result of these 
sessions were made before the independent critiques. 

In general, however, the flowchart was very well received and needed little 
reworking. One expert commented that the flowchart with hyperlinks is “a wonderful, 
simple, and useful product.” Another expert expressed his hope that the product will be 
included in the cost boot camp program at ASC, which is an introductory training course 
for new cost analysts. All experts supported the flow of the process as represented by the 
flowchart and ensured the content of the linked knowledge pages was accurate. After all 
six experts reviewed the flowchart and its linked knowledge pages and provided 
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feedback, the next phase of critique sessions were set up with experts independent of the 
knowledge elicitation portion of this study. 

Phase 2 - Critique by an Independent Set of Experts. In order to “filter” the 
knowledge contained in the flowchart, independent experts need to review and critique it. 
The set of independent experts contacted for this phase are recognized top leaders and 
experts in the cost analysis community at Wright-Patterson AFB. All currently serve in 
supervisory positions in the financial management community, are still responsible for 
crosschecking cost estimates, and are involved in the training and development of new 
cost analysts. As a result, these experts met the criteria of the expert checklist and were 
deemed qualified to provide input regarding the usefulness of the flowchart. Table 17 
contains the profiles of these independent experts. 


Table 17. Profiles of independent experts 


■ 

Rank 

Career 

Field 

Education 

Current Job Title 

Years 

Experience in 
Cost Analysis 

1 

GS-14 

Finance 

BS 

Chief, Financial 
Management 


2 

GS-14 

Finance 

BS, MS 

Chief, Cost 
Research 


3 

GS-15 

Finance 

BS, MBA 

Chief, Estimating & 
Scheduling Branch 

warn 

j Mean years of experi 

ience in cost analysis 



The researcher contacted the experts by telephone to schedule the critique 
sessions and notified them that an email message would follow. In addition to the copy 
of the flowchart, a brief background of the research that led up to the development of the 
flowchart was provided to the independent experts in that email message. The 
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background did not include the names of the experts interviewed in knowledge elicitation 
sessions. Rather, it provided the independent experts with a general understanding of 
how the flowchart was developed and its intended purpose. 

Then, interview sessions were conducted with the experts at their offices. Each 
session was very relaxed and unstructured, and lasted no more than 1 hour. The 
independent experts were asked to critique the flowchart in five main areas. These areas 
are listed in table 18. In addition, the independent experts were asked to provide any 
other feedback that they deemed important with respect to the flowchart. 

Table 18. Areas of crosscheck flowchart critiqued by independent experts 


1. Flow of the process - does the crosscheck process as represented in the flowchart 
agree with your experience? 

2. Accuracy of the knowledge - is the knowledge contained in the linked knowledge 
pages accurate? 

3. Currency of the knowledge - is the knowledge contained in the linked knowledge 
pages current? 

4. Clarity of concepts - is the knowledge represented in a way that is clear and easy 
to understand? 

5. Usefulness - do you feel the guide will be a useful tool to new and inexperienced 

cost analysts when crosschecking cost estimates?_ 


First, they compared the flow of the chart to the way they understand and perform 
the crosscheck process. Each expert felt that the flow was consistent with the proper way 
to perform a crosscheck on a cost estimate. One expert did point out, however, that in 
reality the crosscheck process is often abbreviated due to time constraints. Fie further 
emphasized that this was an unfortunate reality and ideally a crosscheck should be 
performed as represented in the flowchart. 

Next, the experts checked the knowledge contained in the flowchart to ensure 
accuracy. Based on their experience and knowledge, all experts felt the knowledge 
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contained in the flowchart was accurate. One expert provided a different perspective of 
the engineering assessment crosscheck methodology. In addition to the knowledge 
contained in the flowchart, the expert suggested extending the description of an 
engineering assessment to include technical experts outside of the program from which 
the estimate originated. This particular expert felt that having an outside engineer with 
expertise in an area related to the program assess the reasonableness of a cost estimate 
added to the credibility and value of the crosscheck. As a result, this suggestion was 
incorporated into the flowchart. 

The third area critiqued was the currency of the knowledge. This required the 
experts to review the knowledge contained within the flowchart to ensure that it reflected 
the current practices in cost analysis. Based on their understanding of current Air Force 
cost analysis initiatives and practices, all experts felt the knowledge did indeed reflect 
current practices in Air Force cost analysis. 

Fourth, the experts were asked to critique to the flowchart in terms of clarity of 
concepts. This is important because in order for a knowledge representation to be useful, 
it must be understandable by others. Although the experts possess extensive background 
in the cost analysis domain that may have helped them understand certain concepts, they 
were familiar with training and educating new and inexperienced cost analysts. This 
perspective allowed them to determine if the concepts contained within the flowchart 
were clear and understandable. All experts expressed the feeling that the flowchart did 
possess clarity. 

Finally, and perhaps most importantly, the experts critiqued the flowchart on how 
useful it may be to new and inexperienced cost analysts in the performance of a 
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crosscheck. Because of their experience in the training and mentoring of new cost 
analysts, the independent experts were able to critique the flowchart in this area. Similar 
to other areas critiqued, the majority of feedback was positive. The independent experts 
felt the flowchart was a “good tool” and that it “presented the knowledge in a neat and 
efficient manner.” They felt the flowchart was a useful tool that could be used not only 
by new cost analysts, but by more experienced cost analysts as a refresher on how to 
properly perform a crosscheck. Since this application had not previously been 
considered, this feedback presented a potentially new application for this product. 

The independent experts also expressed interest in expanding this type of 
knowledge representation method to other processes in both the cost analysis and DoD 
financial management knowledge domains. One expert even asked if any other experts 
had made suggestions for other applications of this type of process. He felt that 
organizing knowledge in this type of representation will facilitate learning and instill 
standardization in critical processes. 

Overall, the critique sessions with independent experts produced positive 
feedback. The experts were excited about the future possibilities of this type of tool and 
the effect it may have on reducing the effects of turnover. They were interested in the 
concept of capturing and converting expert knowledge as a way to support initiatives like 
the LCC/LPI and Knowledge Management efforts in the cost analysis knowledge 
domain. They also expressed an interest in seeing what type of effect this tool will have 
on inexperienced analysts. All were very optimistic that it will indeed help inexperienced 
cost analysts perform crosschecks in a standardized and accurate manner. 
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V. Discussion 


Importance of Findings 

This research is important for four main reasons. First, it presents a new 
application of Knowledge Management (KM). The KM framework presented by Cho 
and colleagues was adapted to this thesis, which then sought to develop a KM process to 
support the sponsoring organization in their efforts with the LCC/LPI project. It is 
hoped that this study has satisfactorily demonstrated that KM can be applied to the cost 
analysis knowledge domain to minimize knowledge loss by capturing and organizing 
knowledge before it’s lost. 

The second reason this study is important is because it presents a KM process 
designed specifically for the cost analysis knowledge domain to generate, organize, and 
develop knowledge. It was also designed to meet the unique research constraints 
presented in the comparative analysis section of chapter 2, which are also common to the 
current acquisition environment in which cost analysts operate. Several possible 
knowledge elicitation and representation techniques were considered for this study. The 
researcher selected techniques that can be used by inexperienced cost analysts to 
generate, organize, and develop knowledge pertaining to an identified task. The 
methodology was successfully demonstrated on the task of crosschecking cost estimates, 
resulting in the creation of a guide containing expert knowledge pertaining to this task. 

Third, the knowledge representation tool used in this study, namely flowcharts 
with hyperlinks to detailed knowledge, is a new application for the cost analysis 
knowledge domain. Many experts that critiqued the crosscheck flowchart expressed their 
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satisfaction with the simple and efficient approach of representing expert knowledge in 
this manner. They believed that the method could be used to represent several other 
processes performed by cost analysts, further adding to the knowledge inventory already 
generated from this study. They also believe that the end product is an interesting, yet 
simple way to facilitate the learning of inexperienced cost analysts in various cost 
analysis activities. The sponsoring organization also expressed the potential of this kind 
of knowledge representation tool in future cost analysis training courses and materials. 

Finally, this study is important is because it produced a usable guide to be used by 
cost analysts when crosschecking estimates. Crosschecks are important because they 
add credibility to a cost estimate. This is very important when a cost estimate is being 
presented to decision makers or reconciled against independent cost estimates prepared 
by the Office of the Secretary of Defense Cost Analysis Improvement Group 
(OSD/CAIG). To ensure that a crosscheck has been completed properly and completely, 
the crosscheck guide created as a result of this thesis can be used as a reference. In order 
for this guide to remain current and useful, a member of the sponsoring organization will 
need to update the guide as needed. 

Limitations and Validity Issues 

While every effort has been taken to enhance the reliability of this study, threats 
do exist. The knowledge researcher has to make several decisions throughout the 
methodology. These decisions include how many experts interview, the tools used for 
the think aloud session (cost estimate and example problem), how much and what 
knowledge to document, and how to organize the steps in the process during knowledge 
representation. Because of the subjective nature of many of these decisions, the 
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background, education, and experience of the knowledge researcher may produce 
different results. However, the methodology was developed from the perspective of an 
inexperienced cost analyst and is designed for use by these types of individuals in the 
field. In addition, the specification of constraints strengthens the reliability because it 
establishes the conditions under which the methodology can be used. 

Although the content of the crosscheck flowchart has been “filtered” by experts, 
this does not mean that it will be well received in the field by inexperienced cost analysts. 
As mentioned by Harrison, “the reliability and validity of a” knowledge based system 
“must continue throughout the lifetime of the system” (Harrison, 1989:326). In order to 
truly evaluate the usefulness of the knowledge representation tool, it must be field-tested 
by end users. This includes having them perform crosschecks on cost estimates using the 
crosscheck guide. 

Field-testing was not feasible in this study due to time constraints. This lack of 
field-testing with end users weakens the internal validity of the study. Having recognized 
experts critique the end product, however, does provide a good indication of its 
usefulness. The results of the critique sessions suggest that the content generated and 
organized pertaining to crosschecking cost estimates is accurate and useful. 

Another limitation of this thesis is the sample of experts whose knowledge was 
elicited for this thesis. Although the sample satisfied the size requirement specified in the 
literature and was reasonably representative of the Wright-Patterson AFB cost analyst 
population, it was limited to one location. This poses a threat to the external validity of 
this thesis and the resulting crosscheck guide. While it is hoped that the findings of this 
thesis is similar to what may be found at other organizations within the Air Force cost 
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analysis community, it is not certain. One way this external validity issue can be 
overcome in future research is by having a mix of experts from different locations in the 
Air Force. This was not possible in this thesis due to funding constraints. 

The final limitation of this research is in regard to scope of the demonstration of 
the methodology. The method was only demonstrated on one task in the cost analysis 
knowledge domain - crosschecking cost estimates. This presents yet another threat to 
both internal and external validity. While demonstrating the methodology on several cost 
analysis tasks or processes would counter this threat, time constraints prevented this 
option. As a result, while it is hoped that the methodology can be used to generate, 
organize, and develop knowledge pertaining to other cost analysis processes, it is not 
certain. 

Future Research 

All of the limitations of this study present opportunities for future research. In 
addition, the sponsoring organization’s satisfaction with this thesis ensures that future 
research efforts will be supported. One potential area of future research involves further 
development of the methodology presented in this thesis. Other knowledge elicitation 
techniques could be explored to generate knowledge pertaining to an identified task. This 
knowledge could then be represented and organized using other knowledge 
representation tools. This type of research would further improve the cost analysis 
community’s ability to generate, organize, and develop the knowledge vital in performing 
its mission. 

Another option for follow-on research could include field-testing and further 
refinement of the crosscheck flowchart. This might include eliciting knowledge from 


75 



expert cost analysts outside of Wright-Patterson AFB to increase the flowchart’s external 
validity, qualitative research documenting the benefit of the flowchart, or surveys 
measuring the acceptance and perceived benefits of the crosscheck flowchart among 
inexperienced cost analysts. All of these possibilities would enhance the validity of the 
methodology and improve the quality of crosschecks on weapon system cost estimates. 

Yet another possibility for future research is the application of this methodology 
to other cost analysis tasks and processes. This would increase the inventory of 
knowledge pertaining to various cost analysis processes. This possibility would also 
further minimize the effect of knowledge loss on mission performance by capturing and 
organizing knowledge pertaining to several different areas in cost analysis. 

Finally, future research could involve efforts to further adapt the KM framework 
to the cost analysis and financial management community within the DoD. This would 
increase the ability of organizations in these knowledge domains to operate smarter and 
continue to succeed in an environment where financial and personnel resources are 
scarce. Research in these areas are essential to the future success of Knowledge 
Management in helping Acquisition Workers, such as cost analysts, procure affordable 
and reliable weapon systems. 
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Appendix A - Instruction Sheet for Think Aloud Method 


1. Good (morning or afternoon), my name is 1 st Lt Ryan Rueve and I am a student at the 
Air Force Institute of Technology conducting research for my master’s thesis. This 
thesis is sponsored by the Aeronautical Systems Center, Acquisition Cost Division 
(ASC/FMC). The focus of this study is to develop a process to generate, organize, 
and develop knowledge for crosschecking cost estimates. There are two research 
objectives for this thesis: 

a) Minimize knowledge loss that results from employee turnover, and 

b) Increase the performance of novice level cost analysts. 

2. All information that you provide during this session will remain confidential and only 
be used for the purpose of this study, as just described. 

3. The method used for this study is called the think aloud method. This method 
requires an expert (you) to think aloud and say everything you that goes through their 
mind as he or she works through a problem. 

4. In a few minutes, you will be given a practice problem to help you understand the 
think aloud method and practice verbalizing your thought process. The purpose of 
the practice problem is not to test your ability to solve the problem, but rather to get 
used to thinking aloud and saying everything you are thinking as you work your way 
through the problem. 

5. While working through the practice problem, please do the following: 

- Say out loud everything you are thinking from the first time you see the task until 
you give an answer 

Talk aloud constantly 

- Don’t think about what you are going to say 

- Don’t explain what you are saying 

6. Are you ready to begin the practice problem? <If yes, administer practice problem> 

7. Do you feel comfortable with the method? Do you understand that you are to say 

everything you think about as you work through a problem? <If yes, proceed to step 

8. If no, practice again or provide clarification> _ 

8. You will now be presented with a real-life cost estimate and asked to perform a 
crosscheck as you normally would if this were a real-world tasking. The estimate you 
will crosscheck is representative of what you might encounter in your daily 
responsibilities as a cost analyst. 

9. While working through the cross-check please do the following: 

- Say out loud everything you are thinking from the first time you see the task until 
you give an answer 

Tal k aloud con stantl y 
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- Don’t think about what you are going to say 

- Don’t explain what you are saying 

If you stop talking, I will ask you to keep talking and thinking aloud as you work 
through the crosscheck. 

10. If you need to take a phone call or use the restroom at any time during this session, 
please do so and note where you left off in your crosscheck. There is no time limit, 
so please relax and take your time to carefully perform the crosscheck in the manner 
that you normally would. You may use any reference materials or computer software 
programs needed to perform your crosscheck. 

11. Once the session is complete, I will answer any questions you may have regarding the 
purpose or nature of this study. You will be contacted within two business days after 
the completion of this session to schedule a follow-up focused discussion session (if 
necessary). 

12. Thank you for assistance on this study, do you have any questions before we begin? 
<If no, then begin recording the crosscheck. If yes, provide clarification> 
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Appendix B - Example of How to Structure Elicited Knowledee 


Prime Mission Equipment, cont. 


Hardware / Tooling 


TYSM 

FY03 

FY04 

FY05 

FY06 

FY07 ( 

FY08 

FY09 

TOTAL 

EMD ' V 
PROD 

5.9 

11.6 

10.1 

6.1 

19 
49.9 j 

o.i; 

68.4: 

57.9 

.35.7: 

176.2; 


Program Phase: EMD 0 Prod 0 
Methodology: 

- PRICE H, Vendor Quotes, ROMs 

• Production: 

- 93% learning curve for developed items 

- Unit cost for purchased items 

- Risk included in Processor Boards and Circuit components 

Phasing: 

- EMD Hardware: 60/40 S-Curve 

- Production Hardware: Lot quantities 


- Expert 1 knowledge 

- Expert 2 know]edge 
-Expert 3 knowledge 
-Etc. 
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Appendix C - Think Aloud Practice Problem 


You have been asked to evaluate a bidder’s past performance record to see if a risk 
adjustment is warranted based on unsubstantiated rumors of large yearly losses. The 
following data has been provided: 


YEAR 

1985 

1986 

1987 

1988 

1989 

1990 

1991 

1992 

1993 

1994 

1995 

1996 

1997 


Budgeted 

Expense 

$623,474 

$604,081 

$611,118 

$541,944 

$572,544 

$577,982 

$475,081 

$472,828 

$689,315 

$889,566 

$873,809 

$846,645 

$813,607 

$ 8 , 591,994 


Actual 

Expense 

$687,429 

$629,394 

$626,430 

$553,044 

$577,080 

$569,238 

$514,729 

$472,754 

$624,081 

$743,077 

$779,972 

$785,596 

$845,158 

$ 8 , 407,982 


Based on this data, is there a problem and if so, what risk adjustment would you 
recommend? 
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Appendix D - Transcript of One Think Aloud Session 


1. Um... ok trying to figure out what the purpose of this is and its obviously to review 
EMD and production 

2. Um... ok I can see the cost team members 

3. And I actually know who these people are so I know right away just based on who it 
is.. .that will usually tell me if I’m going to trust what they’ve done more or not 

4. I mean I’m always going to check... but I know the lead cost analyst has a lot of 
experience so I’m more likely to trust what she has done compared to what someone 
else has done 

5. Probably what I would normally do is see how they define group a and b because 
depending on what kind of system it is 

6. Everybody seems to define what that is a little bit differently and that can have a 
dramatic impact in what that cost should be and what should be included in that cost 

7. Um... group a estimated by the platforms themsel ves 

8. Based on my experience these guys are usually pretty optimistic 

9. And you like to believe everybody is going to be honest but program offices have 
budget constraints so they’re going to do their best to come in budget 

10. Versus generally when someone from staff does the estimate 

11. Um... we’re just looking to be the honest brokers 

12. Ok we got the...uh...description what this thing is 

13. I’m obviously going to look at it to try to figure out 

14. How its structured and also depending on the kind of detail they have in the estimate 
to see if they’ve accounted for everything 

15. Later on I’m just going to go through here to make sure they accounted for everything 

16. At least group b through single contract 

17. Just going to look to see if each of these phases look right based on my experience 

18. Um...looks to me like 

19. More or less they are trying to develop this in three years which seems right 

20. And then roll it out over a year and that seems right 

21. And then the time frames for the various platforms for the install doing the install for 
group a seems 

22. Based on my experience fairly correct 

23. And then I’m seeing the production quantities that 

24. From a quantity standpoint.. .its not a heck of a lot of quantity so 

25. Obviously the contractor is going to have to charge a larger profit margin because 
they are not making up for it in volume 

26. Then we have quantities and I like to check the charts to make sure the quantity tracks 

27. Um.. .let’s see.. .32 42 43 that tracks 

28. 29 39 31and 57...that tracks 

29. Let’s see...current year dollars equated that makes sense 

30. Group b configuration is the same for all platforms which makes the estimate easier 
to do 

31. Um...initial group a installation occurs 24 months after first kit buy 

32. That seems to make sense based on my experience 

33. Methodology let’s see for hardware and tooling they’re using PRICE H 
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34. And I’ll have to look later but I know price h requires all sorts of detailed parametric 
data like weight volume 

35. For some others they have vendor quotes and ROM 

36. So I’m going to be interested in seeing if they talk about that and how they analyzed 
the ROMs and quotes 

37. Software they used price s...which once again is a parametric model pretty intensive 
on the detail of the data you provide so 

38. And then for hardware and software I’d ask them if they used any kind of cross¬ 
checks again their prime methodologies 

39. Urn.. .integration once again they used parametrics so I’d ask them if they used any 
kind of cross-checks 

40. Urn...then GSE is an engineering assessment so I’ll ask them what they actually 
mean by an engineering assessment 

41. Um.. .then hope the gov’t test center estimate that was done by the test center I’d ask 
them if they have any detail on that 

42. Um.. .cause a lot of times we get burnt on test center estimates 

43. Um.. .then for the interim contractor support they used a CER and once again I’d just 
be interested in where that CER came from 

44. I’m assuming as we go through here they’re going to go through the details on that 
and if not I’ll ask them 

45. Um...when I’m looking at the estimate summary as silly as this sounds I’m looking 
to see if the base year dollars are greater than the TY dollars because sometimes 
they’re not 

46.1 always make sure the quantities are consistent 

47. And...the BY dollars track between the EMD summary and the estimate summary 
and the TY does too 

48. Um.. .pretty much looks like they used a standard S curve to me which seems to make 
sense 

49. Did someone give them the budget beforehand in where they tried to fit the estimate 
in the budget or were they independent in doing their estimate and however it comes 
out it comes out 

50. Looking at the way the dollars are phased by year I’m inclined to believe they just did 
their independent estimate b/c the phasing looks pretty normal to me 

51. Um...from an overall dollar standpoint they got the percentage of total cost for each 
of these cost elements so I’m just looking to see if that makes sense 

52. Um...PME...maybe looks a little low to me on a percentage basis and OGC I’ll have 
to see the detail that seems kind of high it’s third of overall costs 

53. B/c I know if nothing else usually general officers and higher level people have a 
calculator there and I always make sure things track and 

54. If they don’t they imply that everything you do isn’t very good 

55. Um...SEPM seems kind of low 

56. Once again I’ll look at their data to see if the factor they used makes sense 

57. I’ll go back and look at their methodology here 

58. For support equipment they used a factor so I think I’d ask them if they had a better 
idea rather than use a factor what that support equipment is 
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59. They got like 7% of the total for support equipment here and to be as little as 1 or 2% 
or 14 or 15% 

60. Um...prime mission equipment 

61. Pull out the production and EMD summary at the top along with the system 
description so I can look at these as I look through the other stuff 

62.1 think I’d ask them why they broke tooling out separately 

63.1 always just believe it’s inherent in the hardware so I’d ask them if there is any 
particular reason why they called out tooling 

64. Pretty much the definitions track to what you see in the price models 

65. And...I see that software is about 20% of the prime mission product 

66. Which tells me it’s not a heck of a lot of software 

67. So I will be interested in looking at the lines of code and stuff 

68. They told me their using a 93% learning curve and they never explained where that 
learning curve came from 

69. Um...based on my experience that seems in the ballpark but I’d be curious as to 
where that learning curve slope came from 

70. Um...I know price has done some historical reports 

71. Um...the ACEIT database might have some stuff in there 

72. Tecolote has done some.. .different studies so I’d see if they or RAND had something 

73. Frankly I’m a packrat so I have my smart book that over time I gather stuff 

74. And I usually try to make sure the cost library gets stuff 

75. But I just have 2 or 3 binders on my shelf of stuff I’ve collected over the years like 
slope data 

76. But typically I would probably go to the cost library first and check the ACEIT 
database 

77. And every time I have an estimate and am looking for detail I will run some kind of 
search using the cost library 

78.1 do a lot of work with the price guys so I trust that they know there stuff and ask 
them what they seem 

79. Phasing just like I thought they’re using an S curve 

80. Which is certainly appropriate 

81. And production is phased by lot size and that makes sense 

82. Um.. .then they talk about that they’ve used risk using the price h monte carlo 
simulation 

83.1 know a lot about price and you can do monte carlo simulation on multiple items 
elements like weight or whatever 

84. So I’d ask them what input variables they ran the simulation on and where they got 
the range of variables from...who it was from an engineer and how the ranges were 
accounted for 

85. And I probably actually try to get a feel to make sure if they were generally using 
consistent percentages for each item 

86. For example if I find out they were using plus or minus 20% on every item that would 
lead me to believe the risk analysis is almost worthless 

87. Really each item in the price model should independently have their own risk ranges 
and if they’re all kind of a normal distribution with the same ranges it kind of defeats 
the purpose of the risk analysis 
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88. I’d be asking them how many lines of code is it what language is it in um... 

89. Here they talk about schedule risk 

90. But like on hardware they did a monte carlo simulation I’d ask them why they didn’t 
use the price s capability to do a monte carlo analysis here 

91. And why they only accounted for schedule risk and not like...difficulty risk due to 
the technology which once again would be I’d be asking them how many lines of 
code language what % of that code is new design what % is new coding 

92. B/c depending on how much is new design or new coding that will determine how 
hard the job is 

93. I’d ask them to get a feel for how many software people the contractor has because 
there is a real problem with software engineer turnover now 

94. So just try to get a feel on how we think turnover might affect this and that would also 
affect their schedule risk here 

95. That’s usually when our programs fail it’s because of software 

96. Another question I’d ask them is based on their estimate how many lines of code per 
hour their estimate implies 

97. Generally my experience says if I see anything above 2 lines a code an hour I’m 
going to think the estimate is suspect unless I get some technical data that makes me 
more comfortable 

98. If they we’re doing like less than 1 line of code per hour I would want to know why 
the task is so difficult because I would expect to see 1 to 2 lines of code an hour 

99. For the average person...depending on who you’re briefing to they don’t know price 
h/price s so you need to make them understand what you’ve done 

100. Um.. .looks like the next chart here is talking about integration 

101. They’ve got the phasing listed here and it doesn’t look like an S curve to me so 
I’d ask them how they phased the integration again at least for the hardware 

102. For the software they tell me based on milestones and that makes sense 

103. Then they told me they did it based on price h/price s 

104. I guess at this point I would tell them based on my experience that price h/price s 
doesn’t calculate integration costs very well and I’d ask them if they’ve done any 
cross-checks because I’ve had pretty poor results with the price models in that area 

105. Systems engineering program management.. .um 

106. Since they are using factors throughout the overall estimate so much I’d really 
want to make sure that the analogous systems made sense to me 

107. And then I would probably ask them given the lightning bolt initiatives where 
they’re really trying to keep systems program management overhead down 

108. And our gov’t oversight there’s been some trends where that is actually reducing 
the percent of the total cost 

109. Because from what I’ve seen recently if you’re using a historical database even 
though it’s from 1998 

110. The bulk of the programs in that study are a lot older than that 

111. I would tell them that my gut tells me that using a factor out of that study is going 
to give them a factor higher than it needs to be 

112. But certainly there is nothing wrong with using a percent of.. .a percentage factor 
methodology 

113. I think maybe they did execute it properly 
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114. They used a yearly percentage of prime mission equipment for phasing 

115. But probably what I would do is turn dollars into equivalent bodies 

116. To see if the body count per year makes sense because a lot of times when you do 
that it doesn’t make sense 

117. Moving on to contractor system test and evaluation 

118. Boy that factor seems awfully high that they used in EMD 

119. So I’d be looking at the programs they used again 

120. But I’d sure like to understand the planned internal test program because that 
seems awful high 

121. And typically I don’t see any ST&E broken out separately in production 

122. And it’s not very much here only 5% 

123. But I’d ask them what they expect to be doing in production 

124. Ok moving on to training 

125. As I would expect it’s a pretty low dollar amount 

126. Once again I mean like in this case they only used 4 analogous programs versus 
using 15 to 17 on the other ones 

127. I would want to know what caused them to throw the other ones out 

128. But certainly the factors are about what I would expect 

129. And once again based on.. .I’d want to see if they have any kind of feel for what 
kind of actual training is going to occur on this program 

130. Um.. .because are they going to have to set up a class are manuals going to have 
to be done or is it just training the maintenance guy how to do maintenance on there 

131. That would lead to me to at least have confidence on maybe which programs they 
picked as analogies 

132. Moving on to data 

133. The percentage total for data although its low in general its still higher than I 
would expect 

134. Pretty much anymore we don’t ask for data 

135. We just take contractor format 

136. But even though it’s a low dollar item I still think it’s higher than it needs to be 

137. Uh... support equipment 

138. I get a real concern when I see a factor being used for support equipment 

139. Because it’s so program dependent and once again their doing analogies using 
these studies 

140. And see if they had any clue as to what support equipment is going to be required 

141. And see if there is any chance of getting some sort of equipment list instead of 
using factors 

142. Interim contractor support... 

143. They used a CER based on the EF-111 from 1994 which is pretty dam old 

144. I would think that looking through ACEIT or Tecolote or something else 
that...you can do better than using a CER that was calculated back in 1994 

145. Interim contractor support is very contract dependent too 

146. So.. .I’d try to get a feel for what ICS.. .1 mean what their plans were on this 
program 

147. Ok moving on to initial spares 
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148. It’s another one that’s very dependent on what...in this case pretty much...pretty 
much program specific 

149. In this case they used the EF-111 and my question to them would be why did they 
use just that one program 

150. And why they used...especially when using a factor...why they used a single 
point that.. .concerns me 

151. Ok other gov’t costs which when I initially looked through here I thought were 
kind of high 

152. And although no one ever calls it risk I know in ECO a lot of time people hide 
their risk money there 

153. And if I go look I can see they got in TY dollars almost $12 million in ECO 

154. Which looks to me to be less than 10% 

155. Looks to me to be like 7 and a half percent for ECO of the overall total 

156. Typically when I’m doing any EMD estimate I’m looking for 10 to 15% 

157. I mean what makes this program.. .what it really implies is more stable 
requirements and its less risky 

158. Because I would need a compelling argument to use anything less than 10% of the 
total 

159. Ok they go into to detail on the other ones 

160. So let’s just look at their detailed explanations and...I mean...this is 

161. ECO...ok they say they used 10% for EMD but I would ask them what they 
applied that against 

162. So once they prove that they applied it against the right base 10% is still at the 
lower end of the range 

163. So I’d ask them why they didn’t go higher 

164. 5% for production is pretty standard 

165. Once again I mean with the factors they used based on history is pretty much the 
standard 

166. They didn’t go with the higher risk and I’d ask them why they don’t think it’s a 
higher risk program 

167. Ok.. .they do tell me how they calculated it.. .PME plus other contract costs 
(OCC) 

168. Um.. .let’s see if they should have applied that to OGC also 

169. I think the one thing they didn’t include... which I myself would have probably 
included.. .is gov’t tests 

170. So I’m thinking the base they applied ECO against is maybe not correct 

171. Generally in the past I apply it to everything other than like mission support 

172. And they pretty much disregarded all of the OGC items to apply ECO against 
which is why when I calculated the factor before I saw theirs it didn’t jive 

173. So just that alone.. .the factor itself that they used maybe ok although it implies a 
normal standard program 

174. Um...moving on to mission support 

175. I have no problem with this 

176. They basically... the organization this is in they just used a historical factor 

177. And adjusted for the future as they view it today 

178. So that makes sense to use the history 
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179. Uh...gov’t furnished equipment 

180. Test missiles and MILSTRIP...ok 

181. Methodology...ok right away they told me they based it on technical judgement 
which means it’s just a guess 

182. Urn...for the missiles...they’ve got it looks to me like some actual cost history so 
they basically used the current cost of the missiles and I have no problem with that 

183. But I think I would double check just really to scrub them to make sure there 
wasn’t another source for the MILSTRIP portion 

184. Moving on to gov’t test and evaluation 

185. I understand they have the test organization that is going to be doing the testing 
and provide the estimate 

186. But I would see if they can provide me with any detail as to how these guys came 
up with their number 

187. But my experience is that these guys are generally too optimistic 

188. So I would want to have some understanding as to the detail behind their estimate 

189. And gov’t test is another area where we seemed to get burnt also 

190. First destination transportation... 

191. Their just using the standard manual factor for production in EMD 

192. Lessons learned...um 

193. I would obviously ask them when did they get their schedule 

194. How much did it impact them 

195. Did they actually have the detail they needed to do their schedule by the end here 

196. I just would want to understand the impact and see if they had confidence in the 
schedule they do have 

197. Because these program managers will often keep changing things 

198. We need to make sure these lessons learned get out to people 

199. Of the tasks I think its going to take me to get the estimate done 

200. I know I’m going to need a firm PMD 

201. I know I’m going to need to have a schedule there 

202. And so when I build my schedule I know when each of these things need to be 
done 

203. And I’d almost anymore make the program manager sign a contract with me that 
says I’m going to get these things or he agrees with my schedule 

204. And that there is a day for day slip for each day I don’t get things 

205. Or if they change things later it all gets re-done 

206. And so pretty much I’ve learned that unless you have that agreement up front and 
they understand that things will never be in contract and they will always try to 
change things on you 

207. Then their last point here we have a real database problem around here with 
relevant cost history 

208. So it doesn’t surprise me that they talk about.. .this RW calibration study that they 
used for avionics is out of date 

209. Because RW doesn’t even exist anymore 

210. I know that study is from like 1988 which certainly doesn’t tie to the way avionics 
are now 
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211. What I would really ask them is if they used this calibration study in calibrating 
the price models 

212. And if so why they felt that was better than just using the standard defaults 
especially given how old this study is 

213. I would ask them if they had any better or more current specific programs where 
they could have gotten a little better calibration 
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Appendix E - Sample of Actual Documentation of Elicited Knowledge 


Prime Mission Equipment, cont. 

Hardware / Tooling 

TY$M FY03 FY04 FY05 FY06 FY07 FY08 

EMD 5.9 11.6 10.1 6.1 1.9 0.1 

prod |.; 499 68.4 

Program Phase: EMD 0 Prod 0 
Methodology: 

- PRICE H, Vendor Quotes, ROMs 

• Production: 

- 93% learning curve for developed items 

- Unit cost for purchased items 

- Crosschecks for slope: EF-111 SIP Source Selection Bluebook (92%), RW 
“standard” avionics (92%), Sanders info for processor boards/avionics (92%) 

- Risk included in MWS ECU Processor Boards and CPCU Processing 
Boards using Price H Monte Carlo simulation 

Phasing: 

- EMD Hardware: 60/40 S-Curve 

- Production Hardware: Lot quantities 


FY09 TOTAL; 

357; 

57.9 176.2 


- Learning curve of 90-95% ok, but need to cross-check against analogous programs and various 
databases 

- ACEIT database, PRICE, RAND, and cost library all have information on learning curves 

- Where did T i come from (vendor quote or ROM)?—if from PRICE IL then that’s OK 

- Engineering input needed on EC slope and T1 

- Phasing 

- S curve appropriate for EMD 

- Lot quantities appropriate for production 

- Monte carlo simulation variables need to be specified 

- input variables and ranges used for variables needs to be accounted for 

- Don’t use a parametric methodology to cross-check a parametric methodology 

- Look at actuals of similar programs to compare against methodology for hard ware/too ling 

- When using vendor quotes, estimator needs to look at track record of contractor 

- Compare past quotes to contracts to see what kind of adjustment is needed 

- Phasing 

- Check 50/50 point on past analogous programs to see if phasing was skewed one way or the 
other 

- If so, then consider adjusting phasing to match historical trends 
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Appendix F - Sample Follow-up Questions 


Directions : Please answer the following questions in the context of performing a 
crosscheck on a cost estimate that is pre-EMD and still fairly conceptual. The questions 
are grouped according to the general areas that they address._ 


1. Overall, when crosschecking an estimate, what “red-flags” do you look for? Once 
you find some of these “red-flags,” how do you address these areas? 


Risk 


2. How do you quantify risk based on the program schedule (as a means of reality 
checking the risk included in the estimate)? 

3. What percentage of total cost is high for software and how does that affect risk? 

4. What is an acceptable range of percentages for schedule risk? What needs to be 
shown to support a schedule risk figure? 

Groundrules/Assumptions/Cost Team 


5. What standard items do you expect to see in the groundrules/assumptions portion of 
an estimate? If those items are not there, what do you do or need to see to reconcile 
this omission? 

6. Does your crosscheck of an estimate differ based on the cost team that prepared the 
estimate (i.e. SPO vs. cost staff)? 

Sources Used for a Crosscheck 


7. If no analogous systems are available for an analogy, how are the assertions and 
figures in an estimate supported? 

8. What is the hierarchy of sources used in a crosscheck (i.e. cost library, RAND, 
ACEIT, etc.)—in other words, where do you go first and why? 

9. When performing a crosscheck, how do you resolve discrepancies between sources 
(e.g. between engineer assessment and history, between databases, etc.)? 

10. If a factor that differs from ASC historical factors are used in an estimate, what do 
you need to see to accept this deviation from historical factors? 
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11. What is the proper way to analyze a ROM? 


Cost Drivers 

12. When checking the percentage of total cost for each cost element, what are generally 
acceptable ranges based on your experience? 

Phasing 

13. When is S curve phasing appropriate? 

14. When are other types of phasing appropriate? 

Miscellaneous 


15. What base should the ECO factor be applied against? 

16. In your experience, what impact does production quantity have on an estimate in 
terms of profit margin for the contractor? 

17. When using learning curves, what quantity is necessary to justify the claim of a 
learning affect? Does this differ depending on the type of system being acquired 
(satellite, aircraft, avionics, laser, etc.)? 
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Appendix G - Printout of Crosscheck Guide 

A Guide for Crosschecking Early Phase Cost Estimates 

An Expert's Approach 


Prepared by: 

1st Lt Ryan J. Rueve 

Graduate Acquisition Management Program 
Graduate School of Engineering and Management 
Air Force Institute of Technology 





Click here to start using this crosscheck guide 
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Instructions to the User 


Generali 

This guide uses a flowchart to represent knowledge captured from experts pertaining to the 
process of crosschecking cost estimates. The flowchart illustrates the process step-by-step 
for the user and contains detailed information about each step through the use of hyperlinks. 
Hyperlinks are shortcuts to documents, worksheets, or other sources found on a network 
server, intranet, or the Internet. When you click the cell that contains a hyperlink, Microsoft 
Excel opens the file stored at the linked location. 

Accessing Knowledge Unked to Flowchart Symbolsj 

Flowchart symbols that have knowledge linked to them are outlined with a dark border. To 
access the knowledge linked to a given symbol: 

- move the cursor over the symbol you desire to obtain more information about 

- click on the symbol 

These steps will take you to the linked pages pertaining to the step in the process about which 
you need more knowledge. Once you are at one of these pages, if there is any example chart 
or item that is hard to see, simply enlarge the percentage view at the top of Microsoft Excel or 
select print preview from the toolbar and then zoom in on the object you desire to see. 

To return to the main flowchart once a step in the process is complete, click on the 'back to 
main flowchart' arrow found at the bottom of each detailed sheet. This hyperlink will then 
return you to the point on the main flowchart corresponding to the step where you left off in 
the crosscheck process. 

A^licationsj 

This guide is only applicable to cost estimates prepared before Milestone III in the 
acquisition process and that cover the EMD, Production, and/or O&S phases of a program. 
The knowledge captured for the content of this crosscheck guide only pertains to cost 
estimates developed in early phases of the program where actual cost data is typically not 
available, but technical data exists. This generally corresponds to phase 0, Concept 
Exploration (CE); phase 1, Program Definition and Risk Reduction (PDRR); and early on 
in phase 2, Engineering & Manufacturing Development (EMD), in the acquisition process. 

As a result, this guide should only be used to crosscheck cost estimates that were 
prepared before Milestone III in the acquisition process, and in the absence of 
actual cost data (which allows the analyst to perform a more thorough crosscheck). 


Flowchart Symbolsj 


c 


The oval is used to represent the beginning or end of a process. For this 
guide, it represents both the start and completion of a crosscheck. 
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The rectangle is used to represent an action, or step, in the overall 
process. More information about a particular step in the process can be 
obtained by clicking on that rectangle (as indicated above). 



The rounded rectangle is used to represent alternate actions in the 
process. Rounded rectangles are most often found after a decision 
point in the process and often require the user to perform an additional 
step before proceeding to the next step in the overall process. More 
information about a particular alternate action can be obtained by 
clicking on that rounded rectangle. 



The diamond is used to represent decisions that need to be made or 
questions that need to be answered before the user can proceed to the 
next step in the process. The various decisions/answers available to the 
user are indicated on branches that lead to an action or alternate action. 


Flow lines show the sequence of the actions contained in the overall 
process. The direction of 'flow' for a process is indicated by the 
>- flow arrow . The flow lines lead the user to the next action or alternate 
action that needs to be taken after a decision has been made, question 
has been answered, or preceding action has been completed. 


Arrows are hyperlinks that allow the user to move to different pages 
within the crosscheck guide. The destination of the arrow is indicated in 
bold words (as seen below). Now, click on the "Crosscheck 
Flowchart" arrow below to access the flowchart of the crosscheck 
process. 



Back to title page 


Crosscheck 

Flowchart 
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Follow flow line and repeat until 
all major cost elements have 
been crosschecked 


Perform crosscheck 
using alternate 
methodology 


By what % 
does CC differ 
from original? 


Differs by 
more 
than +/- 
15 % 


Differs +/- 
10 - 15 % 


Differs by 
less than 
+/- 10 % 


Results acceptable, 
but causes of 
differences should be 
explored 




enable based on 


Difference 
resolved or 
justified? 


Perform crosscheck 
on major cost 
elements using rules 
of thumb 


Compare result of 
crosscheck to 
original cost estimate 
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Crosscheck 

Complete 








Acquisition Phase of the Program 


As stated in the instructions for this guide, the acquisition phase of the program will 
determine whether or not this guide should be used. When reviewing the introductory 
information about the cost estimate being crosschecked, find the necessary information 
to answer the question immediately following this step. 

This information is usually found in the prefatory slides of a cost briefing where 
basic information about the program is provided. 



PHASE0 I PHASE I 


Determination 
of Mi salon 
N<hh* 


CONCEPT 

EXPLORATION 


PROGRAM 
DEFINITION A RISK 
REDUCTION 


PHASE II 


ENGINEERING 4 
MANUFACTURING 
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PHASE III 


PRODUCTION. FIELDING, 
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Proauction or 

Conduct 

Gogin a Now 

Engineering 4 

yielding/ 

Concept 

Acquisition 

Manufacturing 


Studies 

Program 

Development 

tL _ 


* Source: Introduction to Defense Acquisition Management^ 4th Ed. Fort Belvoir: Defense Systems 

Management College Press, Jun e 1999. _ j _ 

This guide is only applicable to cost estimates prepared 
before Milestone III in the acquisition process and that cover 
the EMD, Production, and O&S phases of a program. 
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The scope of the cost estimate refers to the extent of the acquisition process covered 
in the estimate (i.e. EMD, Production, Life Cycle Costs, etc.)- The scope of the cost 
estimate lets the decision maker know how advanced the planning and definition are 
for the program. 

The crucial part about this step is ensuring that the scope of the estimate is stated accurately. 
If the scope of the cost estimate is EMD only, then ensure only EMD cost estimates have 
been included in the estimate; if the scope is EMD and production, ensure that the estimate 
covers both EMD and production; etc. 


Important Links: DoD 5000 Regulations 




Program Description 


The program description provides a top-level description of the system being estimated. 
This section of an estimate provides the decision maker with a technical overview 
of the system and an idea on how much risk is present in the estimate. The descrip¬ 
tion should address whether or not the technology is proven, commercially available, or 
unproven. The more unproven the technology, the greater the potential for variance 
between the crosscheck and the original estimate. The program description also 
provides the decision maker with an idea about how solid the crosschecks need to be 
to support the risk inherent in the technology being used. An example of a program 
description can be found below. 


XYZ Description 



The XYZ system concept is based on the AFRL LIFE (Laser IRCM Flyout Experiment) 
closed loop laser jamming demonstration architecture. It consists of a 2-color IR Missile 
Warning System providing warning and accurate location of an incoming missile to a 
central countermeasure processor. The central processor controls one or more turrets that 
will locate and track the missile and then jam the missile seeker with a laser, causing it to 

break-lock or miss the aircraft. 


Ensure all elements described in the system description have been estimated and are 
contained in the cost estimate. In addition, the program description should reflect the 
current state of the system. This is very important because early on in the acquisition life 
cycle it is common for the user to change requirements which may cause a change in the 
system description. Be sure that the estimate being crosschecked was based on the current 
system description. 



Back to main 
flowchart 
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Ensure that the groundrules and assumptions under which the estimate was developed 
reflect the current state of the program. At a minimum, the groundrules and assumption 
portion of the estimate should include the following items: 


- The Base Year of The money 

- The inflation indices used to convert BY$ to TY$ (i.e. 2001 OSD, 2001 Air 
Force, etc.) 

- The life cycle years covered by the cost estimate (i.e. FY 2002 to 2015) 

- What requirement document the estimate is based on (Cost Analysis Require¬ 
ments Document, Operational Requirements Document, etc.) 

- The type of contract that is anticipated to be awarded to a prime contractor 
(i.e. Fixed Price, Fixed Price with Economic Price Adjustment, etc.) 

- The type of support concept anticipated for the system (Organic vs. Contractor) 


G&A are extremely important because requirement changes are common when a program 
is in the early stages of the acquisition life cycle. When this happens, the groundrules 
and assumptions upon which a cost estimate is based may change. As a result, you 
must ensure that the groundrules and assumptions upon which the current estimate was 
based are still applicable. Groundrules and assumptions can be checked by comparing 
them to the current Cost Analysis Requirements Document (CARD) if it is available, or 
other system requirement documents such as the Operational Requirements Document 
(ORD) or Mission Need Statement (MNS). 
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Agreement between Cost Estimate and Program Schedule 


Agreement between the cost estimate and the program schedule is essential in the cross¬ 
check process. This step involves ensuring that the estimated costs for the various 
components of the system match the planning set forth in the program schedule. 

Assuming that briefing charts have already been made, this step is accomplished by 
taking the following actions: 

1) Locate the cost estimate summary charts and the program schedule in the cost estimate. 

2) If the charts are not contained in the estimate material provided, consider adding them 
to the cost briefing. Summary charts serve as a quick reference and provide the decision 
maker with a top-level summary of the cost estimate. 

3) Once the necessary information is in hand, ensure milestones are clearly identified in 
the program schedule. 

4) For EMD, ensure dollars match the level of effort contained on the program schedule. 

5) For production, ensure dollars match the buy profile of units to be procured. For example, 
if the schedule indicates that 20 units are to be procured in FY03, ensure that dollars have 
been estimated in FY03 to cover the procurement of 20 units. 

An example can be found on the following page. 
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Program Schedule 


Calendar Year 



According to the program schedule, EMD is supposed to end in 
March 2008. Therefore, the EMD portion of the cost estimate 
should extend out to 2008. In this example, it does and the 
charts are in agreement. 


EMDSumri\ary 
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If the program schedule and cost summary do not agree, the source of the problem needs 
to be identified and corrective actions taken. This will involve referring back to the step in 
the process where groundrules and assumptions are checked. If a requirement change 
has caused the schedule to change, then the estimate needs to be corrected to reflect the 
current state of the program. Regardless of the action taken, the ultimate goal of this step 
is to ensure that the current cost estimate is in agreement with the current program schedule. 
Please keep in mind that several "what if" scenarios may be estimated, leaving different 
estimates based on different requirements and assumptions. As a result, you must 
ensure that each "what if" scenario has the correct corresponding estimate with it. 
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Mathematical Computations 


Although it seems obvious that mathematical calculations should be double-checked, 
failure to complete this step has resulted in intense scrutiny by senior decision makers. 

1. Ensure figures sum correctly - this is critical because decision makers frequently add 
numbers to check the credibility of the estimator; if a mistake exists, the credibility of the 
cost team is damaged. As a result, it is crucial that all elements of the cost estimate 
have been summed correctly. 

Example 


For Official 1st Only 


1.1 System Level Integration & Test (32.78 FY1999 $M) 


WBS Definition 

This is contractor system-level compatibility testing. This cost excludes fee and COM. 

Estimating Methodology 

System-Level l&T Cost = XI * X2 

XI = Labor Rate Per Person Per Year = $205K (FY1999) 

X2 = I&T Total Headcount =160 
Rationale 

It is calculated by multiplying headcounts by annual labor cost per year. All headcounts are separated according to 3600 and 3020 funding. 
The time span for the compatibility testing for ABC should be significantly shorter than for the EFG program. Therefore the headcounts 
were scaled back to account for the shorter time span. Labor rate is the average of contractor labor rates from DCMC forward pricing rate 
agreements. 


FY02 FY03 FY04 FY05 FY06 FY07 


Satellites 1 & 2 
Satellite 3 
Satellite 4 
Satellite 5 


15 

13: 


25 

12 


40 


10 


40 


50: 



2. Check the inflation rates - check rates used for this estimate and ensure they match those 
published on the SAF/FMC website. Then, ensure the rates were applied correctly to the 
cost estimate. A general rule of thumb that applies to the majority of cost estimates is that 
TY$ should always be greater than BY$. However, if there are significant actual costs 
incurred in prior fiscal years for the program, this rule of thumb will not hold and BY$ may be 
greater than TY$. 


105 










Important links: SAF/FM Website 


To obtain the latest inflation indices , click the above link and then 

- Click on the "FMC" tab at the top of the page 

- Click on the "Inflation" button on the left side of the SAF/FMC page 

- Scroll down and locate the link to the latest inflation indices 

To obtain the inflation tutorial add-in for Microsoft Excel: 

- Click on the "FMC" tab at the top of the page 

- Click on the "Inflation" button on the left side of the SAF/FMC page 

- Scroll down until you find the hyperlink to the latest version of the inflation tutorial 

- Click the link, select "save this program to disk," and save it to your computer's 
desktop 

- Once the download is complete, minimize all programs and double-click the icon on 
the desktop representing the zipped inflation tutorial file 

- When prompted, unzip the file to “c\windows\desktop" and click unzip 

- Once the files have been unzipped, double-click the icon "TutorXX" 

- Bring up Microsoft Excel and click the "enable Macros" button when prompted 

- The inflation tutorial can now be accessed under the "Tools" menu at the top of Microsoft 
Excel 

Once all mathematical computations have been checked, you should have the necessary 
information to answer the next question in the process. This step is complete once all 
mathematical calculations in the estimate are correct. 
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Correct Mathematical Errors 


If errors exist in calculations contained in the cost estimate, go back to the original cost 
estimate and make the necessary corrections. This may involve changing the formulas 
in Microsoft Excel, obtaining the latest inflation indices, or using the inflation tutorial to 
correctly apply inflation. Once the necessary action has been taken, follow the flow line 
from this step and repeat the mathematical check on the estimate to ensure that all 
mathematical calculations have been correctly performed. 
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Determine Primary Methodology Used for Given Cost Element 


Selecting the most effective alternate methodology to use for a crosscheck depends on 
the primary methodology used for a given cost element. Once the primary methodology has 
been identified for a given cost element: 

- Click on the rectangle in the next step of the process that corresponds to that 
methodology 

- Use one of the alternate methodologies found on the linked page for a crosscheck 

For example, if you are crosschecking the hardware portion of prime mission equipment in 
a cost estimate and the primary methodology used was parametric modeling, click on the 
rectangle immediately following this step labeled 'parametric models.' This will take you to 
the detailed sheet on alternate methodologies most effective in crosschecking the parametric 
modeling methodology. 

Cost Element Structure 

Repeat this step for each major cost element in the cost estimate (as indicated by the flow 
line branching off of step 14). The cost element structure for the EMD and production portions 
of cost estimates are program specific and, therefore, may differ from estimate to estimate. 

On the other hand, the cost element structure for the Operation and Support portion of an 
estimate is common across cost estimates. According to the textbook for the Intermediate 
Cost Analysis class (BCE 204): 

“What the WBS is to development and production, the CES is to operating and support. 

A cost element structure (CES) establishes a standard vocabulary for identifying and 
and classifying the O&S costs of a system. The OSD Cost Analysis Improvement Group 
(OSD/CAIG) publishes the Operating and Support Cost-Estimating Guide. This guide 
contains both the generic and weapon system specific CESs for cost estimates requiring 
CAIG review. As with the WBS, the CES is designed to capture as many of the relevant 
costs as practical, but is intended to be tailored to meet the specific system needs.” 

- Excerpt from Intermediate Cost Analysis (BCE 204) textbook 

As a result, perform crosschecks at the major cost element level of the current 
estimate. 


NOTE: 

The major cost elements of the estimate used in the research portion of the development 
of this guide was organized by the following major cost elements: 


PME 

Hardware 

Software 

Integration 

SE/PM 

ST&E 


Training 

Data 

Peculiar Support Equipment 
OGC 

Engineering Change Orders 
Mission Support 


OGC (continued) 

Gov't Furnished Equipment 
Gov't Test & Evaluation 
First Destination Transportation 



Back to main 
flowchart 
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Primary Methodology: Parametric (Models/CERs) 


If the primary methodology used to estimate the costs for a given cost element is parametric 
models, then the alternate methodology used to perform the crosscheck should be (in order 
of preference): 

1} Analogoussystems 

Using the cost library, select analogous systems to use for a crosscheck of the figures 
contained in the original estimate. Criteria for selecting and narrowing the list of analogous 
systems can be found below. 

Criteria for selecting analogous systems/subsystems: 

* If the system being crosschecked is replacing an older system, use the system which the 
new system will replace (e.g., F-22 replacing F-15C, the CF6 engine replacing the TF39 
engine on the C-5, etc.). 

* Use a system with a similar mission—another fighter, bomber, or cargo aircraft; avionics 
system on another similar aircraft; similar engine mission, landing gear system, etc. 

* Use a system with similar software characteristics (particularly for subsystems such 
as avionics). 

** Look at computers used, software language, location of software development 
(same lab developing both?) 

** Is the prime contractor writing the subsystem software as well as the major system 
integration software, or are subsystem and system software being written by 
multiple sources? 

* A system with a similar support concept. 

** For instance, an organically maintained system will require different estimating 
categories (e.g., data procurement, initial spares, lay-in of depot activation equip¬ 
ment) from a contractor supported system where the USAF pays $1000/FH and 
sends all broken assets back to the contractor for repair. 

Criteria for Pairing down group of analogous systems/subsystems: 

* The quantities projected for the new system are significantly different from the quantities 
procured by the possible analogous system. 

** For example, wouldn’t want to use the F-16 or F-15 programs with each having 
procured over 1000 aircraft as an analogous system for the B-2 which bought only 
20 aircraft. 

** Any quantity driven/business base factors (for SE/PM, ECO, etc.) or learning curve 
information from the F-15 or F-16 will probably be outside the relevant range for a 
small procurement such as the B-2’s 20 aircraft. 

* Design differences: although similar mission or the replacement system, components 
or subsystems may not be appropriate to use as an analogous system. 

** For instance, a lot of F-15 historical data on engine cost may be appropriate as an 
analogous system for the F-22 engine, but the F-15 has a federated avionics suite 
while the F-22 avionics suite is integrated; therefore, the F-15 avionics suite is not 
a good analogous system for estimating the projected cost of the F-22 avionics 
suite. 

* Test program: for DT&E and OT&E estimates, try to find an analogous system which had 
about the same amount of technical difficulty and same projected testing profile. 

* Support concept: remove any possible analogous systems with drastically different 
support concept that makes the 2 systems non-comparable. 
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Once analogous systems are selected, compare their costs to the estimate 
generated using the primary methodology. Make a note of the percentage difference 
between the two figures. 

2) Engineering Assessment 

Seek the opinion of the technical experts in the program office regarding the cost estimate 
generated by the parametric model. Also, request that they check the parameters used in 
the parametric model to ensure realism. If the parameters are off or deemed unrealistic 
by the technical expert given the technical complexity of the system, then consider revising 
the parameters used in the model runs. 

It is also recommended that you consult engineers outside the program of the current 
estimate that have a specialty related to the current system. Having outside technical 
experts crosscheck an estimate lends credibility to the cost estimate, especially if they 
are recognized by decision makers. 

3} VendorQuotes 

Contact the prime contractors competing for the contract on the system (or the prime that 
has already been selected) and request quotes for the various component being cross¬ 
checked. This should be accomplished through the program office. The program office 
should be able to obtain quotes for the cost element of interest. Once the quote is obtained, 
compare it to the figure calculated by the primary methodology in the original estimate. 

Also, compare past quotes to actual contract prices for the contractor the quote is being 
obtained from to determine if an adjustment is necessary. 

4) ROM 

Obtain a rough order of magnitude from the technical experts in the program office and 
compare it to the original cost estimate. This should only be used as a general guide to 
determine if the original estimate is in the ballpark. 

NOTES 

1. If learning curves are used for the production portion of an estimate, the slope should be 
crosschecked by: 

- Obtaining learning curve information from the ACEIT, RAND, and/or PRICE 
databases pertaining to the nature of the system being crosschecked. 

- Comparing the slope used on the current system to those used on analogous 
systems. 

- Obtaining input from technical experts regarding the realism of the slope being 
used on the current system. 

Any discrepancies discovered by employing any of these three methods should either be 
reconciled or supported by technical reasons for the difference. The T1 used for the curren 
estimate can be crosschecked by looking at actual T1 costs of analogous systems, if 
available, or by engineering assessment. 

2. DO NOT use a parametric model to crosscheck another parametric technique. 
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Primary Methodology: Analogy 


If the primary methodology used to estimate the costs for a given cost element is analogy, 
then the alternate methodology used to perform the crosscheck should be (in order of 
preference): 

1) Parametric Models 

Contact the technical experts from the program office to see if detailed technical data is 
available to support the use of parametric models. If so, obtain the data and compute the 
costs to crosscheck the figure provided in the original cost estimate. Care should be taken 
when obtaining this data to ensure it is free from bias and not too optimistic. One way to 
do this is by having another independent expert review some of the assumptions made 
by the technical expert from the program office. 

2} Engineering. Assessment 

Seek the opinion of the technical experts in the program office regarding the cost estimate 
generated through analogy. Ensure that the analogous systems are indeed analogous 
based on the assessment of the technical experts from the program office. Also, find out 
if any special conditions exist that would cause the current system to vary from the analogous 
systems (i.e. advancements in technology, specialized equipment, etc.). 

It is also recommended that you consult engineers outside the program of the current 
estimate that have a specialty related to the current system. Having outside technical 
experts crosscheck an estimate lends credibility to the cost estimate, especially if they 
are recognized by decision makers. 

3} VendorQuotes 

Contact the prime contractors competing for the contract on the system (or the prime that 
has already been selected) and request quotes for the various component being cross¬ 
checked. This should be accomplished through the program office. The program office 
should be able to obtain quotes for the cost element of interest. Once the quote is obtained, 
compare it to the figure calculated by the primary methodology in the original estimate. 

Also, compare past quotes to actual contract prices for the contractor the quote is being 
obtained from to determine if an adjustment is necessary. 

4 ) ROM 

Obtain a rough order of magnitude from the technical experts in the program office and 
compare it to the original cost estimate. This should only be used as a general guide to 
determine if the original estimate is in the ballpark. 
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Primary Methodology: Vendor Quotes 


If the primary methodology used to estimate the costs for a given cost element is a vendor 
quote, then the alternate methodology used to perform the crosscheck should be (in order 
of preference): 

1) Analogous systems 

Using the cost library, select analogous systems to use for a crosscheck of the figures 
contained in the original estimate. Criteria for selecting and narrowing the list of analogous 
systems can be found below. 

Criteria for selecting analogous systems/subsystems: 

* If the system being crosschecked is replacing an older system, use the system which the 
new system will replace (e.g., F-22 replacing F-15C, the CF6 engine replacing the TF39 
engine on the C-5, etc.). 

* Use a system with a similar mission—another fighter, bomber, or cargo aircraft; avionics 
system on another similar aircraft; similar engine mission, landing gear system, etc. 

* Use a system with similar software characteristics (particularly for subsystems such 
as avionics). 

** Look at computers used, software language, location of software development 
(same lab developing both?) 

** Is the prime contractor writing the subsystem software as well as the major system 
integration software, or are subsystem and system software being written by 
multiple sources? 

* A system with a similar support concept. 

** For instance, an organically maintained system will require different estimating 
categories (e.g., data procurement, initial spares, lay-in of depot activation equip¬ 
ment) from a contractor supported system where the USAF pays $1000/FH and 
sends all broken assets back to the contractor for repair. 

Criteria for Pairing down group of analogous systems/subsystems: 

* The quantities projected for the new system are significantly different from the quantities 
procured by the possible analogous system. 

** For example, wouldn’t want to use the F-16 or F-15 programs with each having 
procured over 1000 aircraft as an analogous system for the B-2 which bought only 
20 aircraft. 

** Any quantity driven/business base factors (for SE/PM, ECO, etc.) or learning curve 
information from the F-15 or F-16 will probably be outside the relevant range for a 
small procurement such as the B-2’s 20 aircraft. 

* Design differences: although similar mission or the replacement system, components 
or subsystems may not be appropriate to use as an analogous system. 

** For instance, a lot of F-15 historical data on engine cost may be appropriate as an 
analogous system for the F-22 engine, but the F-15 has a federated avionics suite 
while the F-22 avionics suite is integrated; therefore, the F-15 avionics suite is not 
a good analogous system for estimating the projected cost of the F-22 avionics 
suite. 
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* Test program: for DT&E and OT&E estimates, try to find an analogous system which had 
about the same amount of technical difficulty and same projected testing profile. 

* Support concept: remove any possible analogous systems with drastically different 
support concept that makes the 2 systems non-comparable. 

Once analogous systems are selected, compare their costs to the estimate generated using 
the primary methodology. Make a note of the percentage difference between the two figures. 

2} Parametric Models 

Contact the technical experts from the program office to see if detailed technical data is 
available to support the use of parametric models. If so, obtain the data and compute the 
costs to crosscheck the figure provided in the original cost estimate. Care should be taken 
when obtaining this data to ensure it is free from bias and not too optimistic. One way to 
do this is by having another independent expert review some of the assumptions made 
by the technical expert from the program office. 

3} Engjneerjng, Assessment 

Seek the opinion of the technical experts in the program office regarding the accuracy of the 
vendor quotes. Have them assess the assumptions made by the contractor and determine 
if, given the technical complexity and unique conditions that may exist, the vendor quote is 
realistic. 

It is also recommended that you consult engineers outside the program of the current 
estimate that have a specialty related to the current system. Having outside technical 
experts crosscheck an estimate lends credibility to the cost estimate, especially if they 
are recognized by decision makers. 
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Primary Methodology: Cost Factors 


If the primary methodology used to estimate the costs for a given cost element is a cost 
factor, then the alternate methodology used to perform the crosscheck should be (in order 
of preference): 

1 ) Factor Studies 

Factor studies are available in the cost library. Call the cost library and request factor studies 
pertaining to the nature of the system (e.g. avionics factor studies for an avionics system) 
Once the relevant factor study or studies have been obtained, compare the factor(s) used in 
the original estimate to the factor(s) found in the study. If a difference exists, calculate the 
percentage effect on the resulting cost figure for evaluation in step 14. Studies conducted by 
SAF/FMC are located on their website. 

Important Link: SAF/FM Website 

To obtain cost factors from the SAF/FMC website, click the link above and then: 

- Click the "FMC” tab at the top of the page 

- Click the "Cost Factors" button on the left side of the SAF/FMC page 

2) Product Center Historical Factors 

ASC has established historical factors that have been computed from past programs. 

These factors are those that have historically held for programs at the given product center. 
Once the relevant historical factor(s) have been obtained, follow the same procedure des¬ 
cribed above. These factors are also available at the cost library. 

3} Analogous Systems 

Using the cost library, select analogous systems to use for a crosscheck of the figures 
contained in the original estimate. Criteria for selecting and narrowing the list of analogous 
systems can be found below. 

Criteria for selecting analogous systems/subsystems: 

* If the system being crosschecked is replacing an older system, use the system which the 
new system will replace (e.g., F-22 replacing F-15C, the CF6 engine replacing the TF39 
engine on the C-5, etc.). 

* Use a system with a similar mission—another fighter, bomber, or cargo aircraft; avionics 
system on another similar aircraft; similar engine mission, landing gear system, etc. 

* Use a system with similar software characteristics (particularly for subsystems such 
as avionics). 

** Look at computers used, software language, location of software development 
(same lab developing both?) 

** Is the prime contractor writing the subsystem software as well as the major system 
integration software, or are subsystem and system software being written by 
multiple sources? 

* A system with a similar support concept. 

** For instance, an organically maintained system will require different estimating 
categories (e.g., data procurement, initial spares, lay-in of depot activation equip¬ 
ment) from a contractor supported system where the USAF pays $1000/FH and 
sends all broken assets back to the contractor for repair. 
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Criteria for Pairing down group of analogous systems/subsystems: 

* The quantities projected for the new system are significantly different from the quantities 
procured by the possible analogous system. 

** For example, wouldn’t want to use the F-16 or F-15 programs with each having 
procured over 1000 aircraft as an analogous system for the B-2 which bought only 
20 aircraft. 

** Any quantity driven/business base factors (for SE/PM, ECO, etc.) or learning curve 
information from the F-15 or F-16 will probably be outside the relevant range for a 
small procurement such as the B-2’s 20 aircraft. 

* Design differences: although similar mission or the replacement system, components 
or subsystems may not be appropriate to use as an analogous system. 

** For instance, a lot of F-15 historical data on engine cost may be appropriate as an 
analogous system for the F-22 engine, but the F-15 has a federated avionics suite 
while the F-22 avionics suite is integrated: therefore, the F-15 avionics suite is not 
a good analogous system for estimating the projected cost of the F-22 avionics 
suite. 

* Test program: for DT&E and OT&E estimates, try to find an analogous system which had 
about the same amount of technical difficulty and same projected testing profile. 

* Support concept: remove any possible analogous systems with drastically different 
support concept that makes the 2 systems non-comparable. 

Once analogous systems have been obtained, calculate the cost factor (if not explicitly 
provided) by dividing the cost estimate for the cost element being crosschecked against 
whatever base the factor is applied against and add one. For example, you are cross¬ 
checking the "Training" cost element and the cost factor used was applied against the Prime 
Mission Equipment (PME) cost. To crosscheck this figure, you would obtain the training cost 
from the analogous system and divide that cost by the PME cost figure for the analogous 
system. Then, add one to the resulting percentage to get the cost factor of the analogous 
system. Compare that factor to the factor used in the estimate being crosschecked to see 
if an adjustment or justification is needed. 

NOTE: If multiple systems are found to be analogous, a composite cost factor 
should be calculated and compared to the current cost factor being used. 


Back to main 
flowchart 
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Primary Methodology: Engineering Build-Ups 


If the primary methodology used to estimate the costs for a given cost element is an 
engineering build up, then the alternate methodology used to perform the crosscheck should 
be (in order of preference): 

1} Analogous systems 

Using the cost library, select analogous systems to use for a crosscheck of the figures 
contained in the original estimate. Criteria for selecting and narrowing the list of analogous 
systems can be found below. 

Criteria for selecting analogous systems/subsystems: 

* If the system being crosschecked is replacing an older system, use the system which the 
new system will replace (e.g., F-22 replacing F-15C, the CF6 engine replacing the TF39 
engine on the C-5, etc.). 

* Use a system with a similar mission—another fighter, bomber, or cargo aircraft; avionics 
system on another similar aircraft; similar engine mission, landing gear system, etc. 

* Use a system with similar software characteristics (particularly for subsystems such 
as avionics). 

** Look at computers used, software language, location of software development 
(same lab developing both?) 

** Is the prime contractor writing the subsystem software as well as the major system 
integration software, or are subsystem and system software being written by 
multiple sources? 

* A system with a similar support concept. 

** For instance, an organically maintained system will require different estimating 
categories (e.g., data procurement, initial spares, lay-in of depot activation equip¬ 
ment) from a contractor supported system where the USAF pays $1000/FH and 
sends all broken assets back to the contractor for repair. 

Criteria for Pairing down group of analogous systems/subsystems: 

* The quantities projected for the new system are significantly different from the quantities 
procured by the possible analogous system. 

** For example, wouldn’t want to use the F-16 or F-15 programs with each having 
procured over 1000 aircraft as an analogous system for the B-2 which bought only 
20 aircraft. 

** Any quantity driven/business base factors (for SE/PM, ECO, etc.) or learning curve 
information from the F-15 or F-16 will probably be outside the relevant range for a 
small procurement such as the B-2’s 20 aircraft. 

* Design differences: although similar mission or the replacement system, components 
or subsystems may not be appropriate to use as an analogous system. 

** For instance, a lot of F-15 historical data on engine cost may be appropriate as an 
analogous system for the F-22 engine, but the F-15 has a federated avionics suite 
while the F-22 avionics suite is integrated; therefore, the F-15 avionics suite is not 
a good analogous system for estimating the projected cost of the F-22 avionics 
suite. 
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* Test program: for DT&E and OT&E estimates, try to find an analogous system which had 
about the same amount of technical difficulty and same projected testing profile. 

* Support concept: remove any possible analogous systems with drastically different 
support concept that makes the 2 systems non-comparable. 

Once analogous systems are selected, compare their costs to the estimate 
generated using the primary methodology. Make a note of the percentage difference 
between the two figures. 


2) Parametric Models 

Contact the technical experts from the program office to see if detailed technical data is 
available to support the use of parametric models. If so, obtain the data and compute the 
costs to crosscheck the figure provided in the original cost estimate. Care should be taken 
when obtaining this data to ensure it is free from bias and not too optimistic. One way to 
do this is by having another independent expert review some of the assumptions made 
by the technical expert from the program office. 
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Perform Crosscheck Using Alternate Methodology 


Using the alternate methodology obtained from the previous step in the crosscheck process, 
calculate a top-level cost to compare to the original estimate. This step may require you to 
elicit information from technical experts or check cost factors used. Regardless of the actions 
taken, the goal of this step is to come up with your own top-level cost estimate that can be 
compared to the original estimate. 

Once you have calculated or obtained your own top-level cost estimate for a given cost 
element, compare that estimate to the original estimate and calculate the difference in both 
dollars and percentage. Once you have done this for a given cost element, return to step 
7 (as indicated by the flow line) and repeat until all cost elements have been crosschecked 
using an alternate methodology. When you have repeated this process for all major cost 
elements, you are ready to proceed and answer the next question in the process for each 
major cost element. 


Back to main 
flowchart 
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Crosscheck Differs bv more than +/-15% 


If the results of your initial crosscheck differ from the original cost estimate by more than 
15%, you need to investigate this difference. If you believe that the difference is warranted, 
then you should support this claim with adequate technical judgments and data. 

"Adequate" support will vary depending on several subjective factors such as who is the 
decision maker for the program, the level of political oversight, the technical expert(s), 
and the nature of the system. As a result, guidance from senior cost analysts should be 
obtained to determine an adequate level of support for the system you are estimating. 

Once you are satisfied that the difference has either been resolved or corrected, you 
may answer the next question in the process to decide if the estimate is ready to proceed 
to the next step in the process. 
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Crosscheck Differs by +/-10-15% 


A difference of 10-15% between the original cost estimate and your crosscheck is normal 
and acceptable. However, if the difference is on the high end of this range this may indicate 
potential problems or require adequate justification as to why this difference exists. 

Although this level of difference between a crosscheck and cost estimate is usually attributed 
to cost estimating error, the difference should still be explored and supported by the 
technical experts of the program and any available data regarding the technology of the 
system. 
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Crosscheck Differs by less than +/-10% 


A crosscheck that produces a top-level estimate that is within 10% of the original estimate 
is ideal. This indicates that the cost estimate can be supported by your crosschecks and 
is therefore a reasonable cost estimate. 
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Perform Crosscheck Using Available Rules of Thumb 


Once a crosscheck using alternate methodologies has been completed, a crosscheck 
using available rules of thumb, or heuristics, should be conducted to further assess the 
reasonableness of the cost estimate. Heuristic crosschecks can also be used in place 
of an alternate methodology crosscheck if insufficient data to support the alternate 
method is available. The heuristics available for the various cost elements are as follows: 

1) PME - Hardware 

Dollars/pound. Divide the total hardware cost of the system by its weight to come up 
with the total cost/pound. Then, compare this cost/pound with that of analogous systems. 

2) PME - Software 

Bodies/year . Convert dollar estimates into equivalent bodies/year using current labor rates 
and compare to history of contractor to ensure realism. This will require information such 
as average salary/programmer, labor hours, and contractor’s historical body/year count. 
This equivalent bodies/year figure can also be compared to analogous systems with similar 
levels of effort to see if it is comparable to figures seen on other programs. If bodies/year 
does not roughly match historical data, then consider making an adjustment. 

Rule of thumb. Obtain data pertaining to lines of code required, programming language 
employed, % of code that is new design, cost/hour, total hours for writing code, and % of 
code that is new. Then, determine how many lines of code/hour the estimate implies by 
dividing the total number of hours by the total number of lines of code - expect to see 1-2 
lines of code/hour, anything above or below this range will require technical support. 

Please note that this rule of thumb may vary depending on several factors. Therefore, 
consult an experienced cost analyst before using this rule of thumb. 

Turnover. Find out how many software engineers the contractor has and what the turnover 
situation is like. If turnover is a problem, then software estimate should account for this in 
the form of schedule risk. In addition, the current employment level can be compared to the 
bodies/year implied by the estimate as described above. 


3) PME - Integration costs 

Contractor history. If it is known who the prime contractor will be, then look at actual 
integration costs of analogous systems handled by that contractor and compare them to 
the current estimate to see if an adjustment is needed. 


4) Systems Engineering/Program Management (SE/PM) 

Bodies/Year. Turn the cost figure into equivalent bodies/year by dividing the annual costs 
by the cost/person. Compare the head count to that of analogous system(s) or to 
other efforts by the contractor (if the contractor is known). 
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Ratios. Take the ratio of this element's cost against the system's hardware cost and 
compare to the same ratio of analogous system(s). 

5) Contractor System Test & Evaluation (ST&E) 

Ratios. Take the ratio of this element's cost against the system's hardware cost and 
compare to the same ratio of analogous system(s). 


6) Training 

Ratios. Take the ratio of this element's cost against the system's hardware cost and 
compare to the same ratio of analogous system(s). 

Level of Effort. Based on the level of effort implied by this system's training concept (i.e. 
type of training, academic materials needed, complexity of training), compare training cost 
of this system to that of analogous system(s) with similar level of effort. 


7) Data 

Ratios. Take the ratio of this element's cost against the system's hardware cost and 
compare to the same ratio of analogous system(s). 

Level of Detail. Based on the level of detail of the data being acquired for the current system 
(i.e. top-level manuals versus level 3 drawings), compare data cost to that of analogous 
system(s) with similar level of effort. 


8) Peculiar Support Equipment (PSE) 

N/A 

9) Interim Contractor Support 

Bodies/Year. Turn the cost figure into equivalent bodies/year by dividing the annual costs 
by the cost/person. Compare the head count to that of analogous system(s) or to 
other efforts by the contractor (if the contractor is known). 


10) Initial Spares 

Rule of Thumb. An accepted rule of thumb is that the cost of initial spares is typically 8% 
of total costs. 
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11) Other Government Costs (OGC) - Engineering Change Orders (ECO) 

Rule of Thumb. ECO is generally 10-15% of all system costs (except ECO) for EMD, 
and 3-7% of all flyaway costs for production. The "flyaway" cost of a system is equal 
to the cost of Prime Mission Equipment + Systems Engineering and Program Mgmt. + 
System Test and Evaluation (for the production portion of an estimate). 


12) OGC - Mission Support 
N/A 


13) OGC - Government Furnished Equipment 
N/A 

14) OGC - Government Test & Evaluation 
N/A 


15) OGC - First Destination Transportation 
N/A 


Back to main 
flowchart 
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The results of the crosscheck using available rules of thumb should support the original cost 
estimate. Any discrepancy or unreasonableness suggested by the crosscheck should be 
either corrected or supported by technical explanation as to why the difference is justified. 
For example, the bodies/year for the software portion of the current estimate should be 
consistent with that of other analogous programs. If differences exist based on the rule 
of thumb techniques, then these differences must be explored. 
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When a heuristic crosscheck indicates a potential discrepancy, you have two options--find 
the problem and correct it or justify the discrepancy with adequate support. The first option 
requires research into why the current cost estimate might differ from what has historically 
held true or from analogous systems. If this investigation uncovers a flaw in some aspect 
of the cost estimate, correct it immediately. If not, then the second option may hold. In this 
case, the discrepancy must be adequately supported. Examples of circumstances that may 
cause an estimate to differ are new technology, some learning effect, competitive advantage 
of a contractor, etc. 
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